Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie

Implementing Gameplay Skills to Increase Eye Contact and Communication for Students with Emotional Behavioral Disorder and Comorbid Disabilities

This study was conducted to examine the effectiveness of gameplay activities using a structured social skills program to increase both eye contact responses and the number of verbal responses during peer relationships for students with comorbid disabilities in a clinical setting. This SEL intervention was modified for children with comorbid autism spectrum disorder (ASD) and emotional and behavioral disorder (EBD). A single-subject, multiple-baseline, across-participants design was used. Participants included adolescents (n = 6) with comorbid disabilities, predominantly ASD and EBD, their board-certified behavior analysts (BCBA), and six behavior technicians. The researcher established and maintained face-to-face reciprocal peer social engagements (communication) and direct eye gaze (contact) with participants in dyad groups. Each session included a 60-minute video of peer social interactions. Direct observations, differential reinforcement of other behaviors (DRO), and pre- and post-Social Skills Improvement System-Rating Scales (SSIS-RS) were examined. Evidence of SEL intervention effectiveness was measured by percentage of nonoverlapping data points (PND). Social validity was measured using the multiple-rater SSIS-RS and intervention fidelity checklists evaluating the Sanford Harmony intervention. Results from data and visual analysis revealed all participants significantly increased their direct eye contact, verbal reciprocity, and social engagements after implementing the Harmony program. In addition, a PND value of 100% was calculated for each dependent variable indicating the Harmony program was a highly effective intervention increasing eye contact, verbal reciprocity, and social engagements for students with EBD and comorbid disabilities

Dissociation and interpersonal autonomic physiology in psychotherapy research: an integrative view encompassing psychodynamic and neuroscience theoretical frameworks

Interpersonal autonomic physiology is an interdisciplinary research field, assessing the relational interdependence of two (or more) interacting individual both at the behavioral and psychophysiological levels. Despite its quite long tradition, only eight studies since 1955 have focused on the interaction of psychotherapy dyads, and none of them have focused on the shared processual level, assessing dynamic phenomena such as dissociation. We longitudinally observed two brief psychodynamic psychotherapies, entirely audio and video-recorded (16 sessions, weekly frequency, 45 min.). Autonomic nervous system measures were continuously collected during each session. Personality, empathy, dissociative features and clinical progress measures were collected prior and post therapy, and after each clinical session. Two-independent judges, trained psychotherapist, codified the interactions\u2019 micro-processes. Time-series based analyses were performed to assess interpersonal synchronization and de-synchronization in patient\u2019s and therapist\u2019s physiological activity. Psychophysiological synchrony revealed a clear association with empathic attunement, while desynchronization phases (range of length 30-150 sec.) showed a linkage with dissociative processes, usually associated to the patient\u2019s narrative core relational trauma. Our findings are discussed under the perspective of psychodynamic models of Stern (\u201cpresent moment\u201d), Sander, Beebe and Lachmann (dyad system model of interaction), Lanius (Trauma model), and the neuroscientific frameworks proposed by Thayer (neurovisceral integration model), and Porges (polyvagal theory). The collected data allows to attempt an integration of these theoretical approaches under the light of Complex Dynamic Systems. The rich theoretical work and the encouraging clinical results might represents a new fascinating frontier of research in psychotherapy

Investigating L1 Arabic EFL Learners’ Interactional and Attentional Processes in Text and Voice Task-based Synchronous Computer-Mediated Communication

While there is a growing body of research considering second and foreign language (L2) learners’ interaction and cognitive engagement in synchronous computer-mediated communication (SCMC), much of the speculation is based on the potential of text chat, with a great deal of uncertainty of how L2 learning opportunities can occur in voice chat. However, voice chat has now become more feasible for this specific purpose, due to the widespread availability of the relevant hardware. In addition, relative to text chat, voice chat has higher social presence and increased social and prosodic cues (e.g. voice tone, stress, intonation), which could promote better understanding and smoother flow of communication. There is, however, a dearth of research on its impact on language learning in comparison to the research on text chat. In a study involving 40 (20 dyads) intermediate-level Arabic learners of English, this thesis attempts to fill this gap by investigating the impact of text chat and voice chat on negotiations and noticing during task-based interactions. The study had a one-shot, repeated-measures design. Stimulated recall interviews were carried out after the completion of the task-based interactions in the two modalities, in order to elicit data on the participants’ noticing of interactional feedback. Follow up questionnaires and interviews were also administered to elicit participants’ perceptions of their learning experience in the two modalities. The findings revealed that voice chat generated more negotiation episodes and incidents of noticing of feedback than text chat. These differences were, however, not statistically significant. Conversely, text chat generated significantly more instances of self-initiated noticing (i.e. self-repairs) than voice chat. Self-repair during text chat, however, tended to focus on spelling. These quantitative findings suggested that, regardless of the SCMC modality, both contexts are equally facilitative for promoting negotiated interaction and noticing of feedback. Moreover, qualitative analysis of the learners’ responses in the debriefing interviews revealed their appreciation for both modalities, implying that both contexts could be incorporated in L2 teaching and learning. In addition, as learners reported that text chat was time-consuming and resulted in incoherent and shallow discourse, pedagogical implications stress that learners need to be prepared for this type of communication, so as to ease the level of completing tasks in text chat, increase their productions and support a more rewarding L2 chatting experience. Additionally, the stimulated recall data offered some methodological implications pertaining to the study of the cognitive process of noticing

THE DEVELOPMENT OF A COUPLE OBSERVATIONAL CODING SYSTEM FOR COMPUTER-MEDIATED COMMUNICATION

Many romantic couples integrate text and computer-mediated communication (CMC) into their relationship dynamics, both for general relationship maintenance and for complex dynamics such as problem solving and conflict. Romantic couple dynamics are interactional, dynamic, and sequenced in nature, and a common method for studying interactions of this nature is observational analyses. However, no behavioral or observational coding systems exist that are able to capture text-based transactional couple communication. The main purpose of this dissertation was to develop an observational coding system that can be used to assess sequenced computer- mediated, text-based communication that takes place between romantic partners. This process included assessing couples’ text communication to determine how verbal and non-verbal communication behaviors are enacted in CMC, modifying an observational coding system, and establishing reliability and validity of the revised coding system. Secondary data was utilized, including 48 logs of romantic couples engaging in problem-solving discussions using online chatting for 15 minutes, where a log of the conversation was saved for future research purposes. For this dissertation, the researcher evaluated the dynamics in these logs to determine if behaviors and sequences were similar to basic romantic relationship dynamics that are present in face-to-face (FtF) couples’ dynamics. The researcher determined that the dynamics between CMC and FtF were similar, and that modifying a couple observational coding system would be appropriate. The Interaction Dimensions Coding System was selected for use and modification for this study, and the training manual and codebook were updated to integrate CMC examples. Multiple avenues of assessing face validity were also pursued and feedback from the coding team and original authors of a couple coding system were integrated into the modified coding system. The modified coding system, IDCS-CMC, was used to code 43 text-based chat logs. A team of 4 coders was trained on the coding system, where they provided ratings from 1 to 9 on each partner for different dimensions of communication behaviors that were observed and they also rated each couple on 5 dyadic categories of relationship functioning. Interrater reliability was assessed throughout the training and independent coding process using the intraclass correlation coefficient. Results indicate that good or excellent interrater reliability was established for the individual dimensions of Positive Affect, Negative Affect, Problem Solving, Support/Validation, Denial, Conflict, and Communication Skills and for the dyadic codes of Positive Escalation, Negative Escalation, Commitment, Satisfaction, and Stability. There were only two dimensions that resulted in fair or poor interrater reliability, which were Dominance and Withdrawal, both of which warrant additional study in how these dynamics are enacted in and coded in CMC. Overall, the IDCS-CMC demonstrated good interrater reliability, and construct validity was established for the coding system in a variety of ways. Construct validity was established by assessing face, content, and convergent validity. Face validity was established by eliciting feedback on the IDCS-CMC from the coding team as well as one of the authors of the system used to inform the development of the IDCS-CMC. Content validity was established by assessing the degree to which the couples in the chat logs engaged in conversations of a similar nature in their real lives, and also by determining the degree to which the couple participants followed instructions to focus on a problem-solving topic during the chats. Convergent validity was assessed by comparing the IDCS-CMC dimensions and positive and negative communication composite scores to a measure of relationship satisfaction. Overall, this dissertation details the process by which a couple observational coding system was developed and tested, and puts forth a methodological tool that can be used to better assess transactional use of CMC by romantic couples by researchers as well as practitioners and therapists

Social attention in young children with autism spectrum disorder: Investigating cross-contextual gaze behaviours, and their relationship to autism severity, cognitive skills and social functioning

Social communication and interaction challenges are characteristic of autism spectrum disorder (ASD). Social attention has emerged to be an important behavioural phenotype in ASD, with accumulating evidence suggesting associations with social functioning and developmental outcomes. However, research gaps remain concerning the nature of social attention, the variability demonstrated across different experimental tasks and social contexts, and the ecologically validity of research methods. This thesis aimed to address these substantive and methodological issues by examining social attention patterns in a young cohort of autistic children, and their age-matched neurotypical peers, across three experimental contexts: 1) a traditional, eye-tracking task with static stimuli, 2) a novel, dynamic eye-tracking task incorporating shared book reading (SBR), and 3) an evaluation of the association in social attention across the two eye-tracking tasks and a play-based social interaction task. In Chapter 2, the influence of circumscribed interests (CI) on social attention patterns was investigated. The results of this study suggested there to be a reduced role for CIs and atypical attention patterns in both social and non-social domains. In Chapter 3, a novel SBR task was developed as a dynamic, ecologically relevant eye-tracking task designed to assess social and joint attention behaviours. Results indicated reduced social and joint attention behaviours, in conjunction with increased attention to non-salient background objects in autistic children. Associations between reduced social attention and poorer social functioning and cognitive skills were also evident in this cohort. In Chapter 4, the social attention patterns of the autistic cohort as measured by the two previous eye tracking tasks were correlated with these patterns in a live, play-based social interaction task between a researcher and the autistic child. Cross-contextual associations in social attention between the social interaction and dynamic tasks, and the dynamic and static tasks were observed. In contrast, there was no significant association in social attention patterns between the social interaction and static tasks. These outcomes contribute new insights into the social attention behaviours of autistic children, and evidence in favor of examining these behaviours in ecologically relevant contexts. They also contribute to evidence associating social attention with autism symptomatology and cognitive functioning. Ultimately, the outcomes of this research may improve our understanding of the needs of autistic children across social, cognitive and adaptive functioning domains