Automatic social role recognition and its application in structuring multiparty interactions

Automatic processing of multiparty interactions is a research domain with important applications in content browsing, summarization and information retrieval. In recent years, several works have been devoted to find regular patterns which speakers exhibit in a multiparty interaction also known as social roles. Most of the research in literature has generally focused on recognition of scenario specific formal roles. More recently, role coding schemes based on informal social roles have been proposed in literature, defining roles based on the behavior speakers have in the functioning of a small group interaction. Informal social roles represent a flexible classification scheme that can generalize across different scenarios of multiparty interaction. In this thesis, we focus on automatic recognition of informal social roles and exploit the influence of informal social roles on speaker behavior for structuring multiparty interactions. To model speaker behavior, we systematically explore various verbal and non verbal cues extracted from turn taking patterns, vocal expression and linguistic style. The influence of social roles on the behavior cues exhibited by a speaker is modeled using a discriminative approach based on conditional random fields. Experiments performed on several hours of meeting data reveal that classification using conditional random fields improves the role recognition performance. We demonstrate the effectiveness of our approach by evaluating it on previously unseen scenarios of multiparty interaction. Furthermore, we also consider whether formal roles and informal roles can be automatically predicted by the same verbal and nonverbal features. We exploit the influence of social roles on turn taking patterns to improve speaker diarization under distant microphone condition. Our work extends the Hidden Markov model (HMM)- Gaussian mixture model (GMM) speaker diarization system, and is based on jointly estimating both the speaker segmentation and social roles in an audio recording. We modify the minimum duration constraint in HMM-GMM diarization system by using role information to model the expected duration of speaker's turn. We also use social role n-grams as prior information to model speaker interaction patterns. Finally, we demonstrate the application of social roles for the problem of topic segmentation in meetings. We exploit our findings that social roles can dynamically change in conversations and use this information to predict topic changes in meetings. We also present an unsupervised method for topic segmentation which combines social roles and lexical cohesion. Experimental results show that social roles improve performance of both speaker diarization and topic segmentation

Building the Bridge in Interaction : Receptive Multilingualism among Finnish and Estonian Speakers

This study investigates receptive multilingualism (RM) among speakers of Finnish and Estonian. Receptive multilingualism refers to interaction in which participants employ a different language from their interlocutors. Mutual understanding in such interaction can be based on mutual intelligibility of the languages, on language acquisition, or on both. Using the methods of linguistic ethnography and conversation analysis, this research focused on the boundary practices of two communities of practice, a Finnish student organization and its Estonian friendship organization. In the studied setting, receptive multilingualism has a special status, defined by a language policy. Triangulation of different kinds of data (conversation data, survey data, field notes and electronic correspondence data) revealed that the language choices in the cross-organizational interaction were a result of a combination of factors: the language skills of the participants, the situation, the number of participants, and different interpretations of the official language policy and its applicability. In the studied context, RM had emblematic value, especially in contexts labeled as “official”. However, to ensure mutual understanding English as a lingua franca was often considered the most effective choice. In the naturally occurring conversations, Finnish-Estonian RM was chosen mostly in multiparty interaction among speakers with asymmetric language skills. In such interaction, the similarity of Finnish and Estonian provided a basis for reaching mutual understanding, but the participants oriented to the languages as mutually intelligible only to a certain degree. That showed in the data 1) as explicit comments on the possibly exclusive nature of using one of the cognate languages, 2) as frequent negotiations of meaning, 3) in how access to the ongoing talk was facilitated, for instance by different kinds of translatory activities, and 4) in how the asymmetric access was made visible and handled by bilingual punning. Speech in the cognate language could be intelligible to the participants due to online recognition of the lexical and grammatical similarities between the languages. Furthermore, the mere awareness of the existence of similarities and even sporadic knowledge about the linguistic equivalences served as an interactional resource. This was apparent in the data 1) in the way that the participants created shared code that mixed the (assumed) resources of Finnish and Estonian, and 2) in how the participants engaged in sharing and co-constructing linguistic knowledge by making bilingual puns. The conversation data also showed that the Finnish-Estonian RM interactions served as a locus of (incidental) language learning.Tutkimuksessa tarkastellaan reseptiivistä monikielisyyttä suomen- ja vironpuhujien välisessä vuorovaikutuksessa. Reseptiivisellä monikielisyydellä tarkoitetaan vuorovaikutusta, jossa eri kielitaustaiset osallistujat käyttävät eri kieliä. Toisen kielen ymmärtäminen voi perustua kielten samankaltaisuuteen, jonkinlaiseen toisen kielen hallintaan tai näiden yhdistelmään. Läheisinä sukukielinä viro ja suomi ovat jossain määrin keskenään ymmärrettäviä. Tutkimuksessa tarkastellaankin sitä, miten yhteinen ymmärrys toisaalta rakentuu kielten samankaltaisuutta resurssina hyödyntäen ja miten se toisaalta syntyy vuorovaikutuksessa, jonka osallistujat osaavat vuorovaikutuksen kieliä eri määriä. Tutkimuksen metodeina ovat lingvistinen etnografia ja keskustelunanalyysi. Monikielisiä kielikäytänteitä tarkastellaan suomalaisen opiskelijajärjestön ja sen virolaisen ystävyysjärjestön jäsenten välisessä kanssakäymisessä. Järjestöjen kielikäytänteissä reseptiivisellä monikielisyydellä on ystävyyssopimuksen perusteella erityinen asema. Eri aineistotyyppien (keskustelu-, kyselytutkimus- ja sähköisen kirjeenvaihdon aineisto sekä kenttämuistiinpanot) triangulaatio osoitti, että kielivalintoihin vaikutti joukko erilaisia tekijöitä: osallistujien määrä, heidän kielitaitonsa, vuorovaikutustilanne sekä osallistujien tulkinnat järjestöjen kielipolitiikasta ja sen sovellettavuudesta käytäntöön. Tarkastellussa yhteisössä reseptiivisellä monikielisyydellä nähtiin symbolista arvoa erityisesti “viralliseksi” määritellyissä tilanteissa. Englantia pidettiin kuitenkin usein tehokkaimpana kielivalintana. Arkikeskusteluista koostuvassa aineistossa reseptiivistä monikielisyyttä hyödynnettiin ennen kaikkea sellaisessa monenkeskisessä vuorovaikutuksessa, jonka osallistujilla oli epäsymmetriset kielelliset repertuaarit. Tutkimuksessa osoitetaan, etteivät osallistujat pitäneet sukukielen ymmärtämistä itsestäänselvyytenä. Se näkyi 1) kommentteina jommankumman sukukielen puhumisesta eksklusiivisena kielivalintana, 2) usein toistuvina merkitysneuvotteluina, 3) osallistumisen fasilitointina esimerkiksi kääntämistoiminnoin sekä 4) kielipeleissä, joiden avulla epäsymmetristä pääsyä keskustelun kieliin käsiteltiin. Sukukielinen puhe saattoi kuitenkin olla ymmärrettävää kielten samankaltaisten piirteiden läpinäkyvyyden vuoksi. Osallistujat käyttivät myös luovasti hyväkseen sukukieltä ja kieltenvälistä suhdetta koskevaa tietämystään. Tietämys tuli näkyväksi 1) vuoroissa, joissa osallistujat yhdistelivät kielten (oletettuja) resursseja, sekä siinä, 2) miten osallistujat jakoivat ja rakensivat kielitietämystään kielellisen leikittelyn sekvensseissä

Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie

Meeting effectiveness and inclusiveness: large-scale measurement, identification of key features, and prediction in real-world remote meetings

Workplace meetings are vital to organizational collaboration, yet relatively little progress has been made toward measuring meeting effectiveness and inclusiveness at scale. The recent rise in remote and hybrid meetings represents an opportunity to do so via computer-mediated communication (CMC) systems. Here, we share the results of an effective and inclusive meetings survey embedded within a CMC system in a diverse set of companies and organizations. We correlate the survey results with objective metrics available from the CMC system to identify the generalizable attributes that characterize perceived effectiveness and inclusiveness in meetings. Additionally, we explore a predictive model of meeting effectiveness and inclusiveness based solely on objective meeting attributes. Lastly, we show challenges and discuss solutions around the subjective measurement of meeting experiences. To our knowledge, this is the largest data-driven study conducted after the pandemic peak to measure, understand, and predict effectiveness and inclusiveness in real-world meetings at an organizational scale

Discourse-level Relations For Opinion Analysis

Opinion analysis deals with subjective phenomena such as judgments, evaluations, feelings, emotions, beliefs and stances. The availability of public opinion over the Internet and face to face conversations; coupled with the need to understand and mine these for end applications has motivated a great amount of research in this field in recent times. Researchers have explored a wide array of knowledge resources for opinion analysis, from words and phrases to syntactic dependencies and semantic relations.In this thesis, we investigate a discourse-level treatment for opinion analysis.In order to realize the discourse-level analysis, we propose a new linguistic representational scheme designed to support interdependent interpretations of opinions in the discourse. We adapt and extend an existing subjectivity annotation scheme to capture discourse-level relations in multi-party meeting corpus. Human inter-annotator agreement studies show that trained human annotators can recognize the elements of our linguistic scheme. Empirically, we test the impact of our discourse-level relations on fine-grained polarity classification. In this process, we also explore two different global inference models for incorporating discourse-based information to augment word-based information. Our results show that the discourse-level relations can augment and improve upon word-based methods for effective fine-grained opinion polarity classification. Further, in this thesis, we explore linguistically motivated features and a global inference paradigm for learning the discourse-level relations form the annotated data. We employ the ideas from our linguistic scheme for recognizing stances in dual-sided debates from the product and political domains. For product debates, we use web mining and rules to learn and employ elements of our discourse-level relations in an unsupervised fashion. For political debates, on the other hand, we take a more exploratory, supervised approach, and encode the building blocks of our discourse-level relations as features for stance classification. Our results show that, the ideas behind the discourse level relations can be learnt and employed effectively to improve overall stance recognition in product debates

Pragmatics & Language Learning, Volume 12

Pragmatics & Language Learning Volume 12 examines the organization of second language and multilingual speakers’ talk and pragmatic knowledge across a range of naturalistic and experimental activities. Based on data collected on Danish, English, Hawaiʻi Creole, Indonesian, and Japanese as target languages, the contributions explore the nexus of pragmatic knowledge, interaction, and L2 learning outside and inside of educational settings. Pragmatics & Language Learning (“PLL”), a refereed series sponsored by the National Foreign Language Resource Center at the University of Hawaiʻi, publishes selected papers from the biennial Conference on International Pragmatics & Language Learning under the editorship of the conference hosts and the series editor, Gabriele Kasper