Measuring, analysing and artificially generating head nodding signals in dyadic social interaction

Social interaction involves rich and complex behaviours where verbal and non-verbal signals are exchanged in dynamic patterns. The aim of this thesis is to explore new ways of measuring and analysing interpersonal coordination as it naturally occurs in social interactions. Specifically, we want to understand what different types of head nods mean in different social contexts, how they are used during face-to-face dyadic conversation, and if they relate to memory and learning. Many current methods are limited by time-consuming and low-resolution data, which cannot capture the full richness of a dyadic social interaction. This thesis explores ways to demonstrate how high-resolution data in this area can give new insights into the study of social interaction. Furthermore, we also want to demonstrate the benefit of using virtual reality to artificially generate interpersonal coordination to test our hypotheses about the meaning of head nodding as a communicative signal. The first study aims to capture two patterns of head nodding signals – fast nods and slow nods – and determine what they mean and how they are used across different conversational contexts. We find that fast nodding signals receiving new information and has a different meaning than slow nods. The second study aims to investigate a link between memory and head nodding behaviour. This exploratory study provided initial hints that there might be a relationship, though further analyses were less clear. In the third study, we aim to test if interactive head nodding in virtual agents can be used to measure how much we like the virtual agent, and whether we learn better from virtual agents that we like. We find no causal link between memory performance and interactivity. In the fourth study, we perform a cross-experimental analysis of how the level of interactivity in different contexts (i.e., real, virtual, and video), impacts on memory and find clear differences between them

Social Affect Regulation and Physical Affection Between Married Partners: An Experimental Examination of the Stress-Buffering Effect of Spousal Touch and the Role of Adult Attachment

abstract: Background: When studying how humans regulate their affect, it is important to recognize that affect regulation does not occur in a vacuum. As humans are an inherently social species, affect plays a crucial evolutionary role in social behavior, and social behavior likewise assumes an important role in affect and affect regulation. Emotion researchers are increasingly interested the specific ways people help to regulate and dysregulate one another’s affect, though experimental examinations of the extant models and theory are relatively few. This thesis presents a broad theoretical framework for social affect regulation between close others, considering the role of attachment theory and its developmental foundations for social affect regulation in adulthood. Affectionate and responsive touch is considered a major mechanism of regulatory benefit between people, both developmentally and in adulthood, and is the focus of the present investigation. Method: A total sample of 231 heterosexual married couples were recruited from the community. Participants were assigned to engage in affectionate touch or sit quietly, and/or engage in positive conversation prior to a stress task. Physiological data was collected continuously across the experiment. Hypotheses: Phasic respiratory sinus arrhythmia (RSA) was used to index the degree of regulatory engagement during the stressor for those who did and did not touch. It was hypothesized that touch would reduce stress appraisal and thus the need for regulatory engagement. This effect was predicted to be greater for those more anxiously attached while increasing the need for regulatory engagement in those more avoidantly attached. Secondarily, partner effects of attachment on sympathetic activation via pre-ejection period (PEP) change were tested. It was predicted that both attachment dimensions would predict a decrease in partner PEP change in the touch condition, with avoidant attachment having the strongest effect. Results: Hierarchical linear modeling techniques were used to account for nonindependence in dyadic observations. The first set of hypotheses were not supported, while the second set were partially supported. Wives’ avoidance significantly predicted husbands’ PEP change, but in the positive direction. This effect also significantly increased in the touch condition. Theoretical considerations and limitations are discussed.Dissertation/ThesisMasters Thesis Psychology 201

Dissociation and interpersonal autonomic physiology in psychotherapy research: an integrative view encompassing psychodynamic and neuroscience theoretical frameworks

Interpersonal autonomic physiology is an interdisciplinary research field, assessing the relational interdependence of two (or more) interacting individual both at the behavioral and psychophysiological levels. Despite its quite long tradition, only eight studies since 1955 have focused on the interaction of psychotherapy dyads, and none of them have focused on the shared processual level, assessing dynamic phenomena such as dissociation. We longitudinally observed two brief psychodynamic psychotherapies, entirely audio and video-recorded (16 sessions, weekly frequency, 45 min.). Autonomic nervous system measures were continuously collected during each session. Personality, empathy, dissociative features and clinical progress measures were collected prior and post therapy, and after each clinical session. Two-independent judges, trained psychotherapist, codified the interactions\u2019 micro-processes. Time-series based analyses were performed to assess interpersonal synchronization and de-synchronization in patient\u2019s and therapist\u2019s physiological activity. Psychophysiological synchrony revealed a clear association with empathic attunement, while desynchronization phases (range of length 30-150 sec.) showed a linkage with dissociative processes, usually associated to the patient\u2019s narrative core relational trauma. Our findings are discussed under the perspective of psychodynamic models of Stern (\u201cpresent moment\u201d), Sander, Beebe and Lachmann (dyad system model of interaction), Lanius (Trauma model), and the neuroscientific frameworks proposed by Thayer (neurovisceral integration model), and Porges (polyvagal theory). The collected data allows to attempt an integration of these theoretical approaches under the light of Complex Dynamic Systems. The rich theoretical work and the encouraging clinical results might represents a new fascinating frontier of research in psychotherapy

Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining two key principles of modality heterogeneity and interconnections that have driven subsequent innovations, and propose a taxonomy of 6 core technical challenges: representation, alignment, reasoning, generation, transference, and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy

Complexity Science in Human Change

This reprint encompasses fourteen contributions that offer avenues towards a better understanding of complex systems in human behavior. The phenomena studied here are generally pattern formation processes that originate in social interaction and psychotherapy. Several accounts are also given of the coordination in body movements and in physiological, neuronal and linguistic processes. A common denominator of such pattern formation is that complexity and entropy of the respective systems become reduced spontaneously, which is the hallmark of self-organization. The various methodological approaches of how to model such processes are presented in some detail. Results from the various methods are systematically compared and discussed. Among these approaches are algorithms for the quantification of synchrony by cross-correlational statistics, surrogate control procedures, recurrence mapping and network models.This volume offers an informative and sophisticated resource for scholars of human change, and as well for students at advanced levels, from graduate to post-doctoral. The reprint is multidisciplinary in nature, binding together the fields of medicine, psychology, physics, and neuroscience

The Social Context of Nonverbal Behaviour

Although nonverbal behaviour has long been a topic of research, it is often studied in isolation from social partners and the social environment. This work presents three empirical chapters that reintroduce the social environment to the investigation of nonverbal cue exchange, focusing on the value of social rewards and the perceptive and affiliative functions of nonverbal communication. Findings reported in Chapter 2 indicate that the subjective value of social rewards changes as a function of social media use saliency. Specifically, thinking about a recent social media post, but not a synchronous conversation, increases the value of social rewards, such that people are willing to forego monetary gain to see a genuine smile. In Chapter 3, I show that although the amount of nonverbal behaviour does not necessarily enhance interpersonal judgement accuracy, accuracy does increase with familiarity, suggesting that people retain and update models of specific social partners. In Chapter 4, I demonstrate that social interactions on video-chat platforms, compared to face-to-face settings, are characterized by reduced interpersonal coordination and increased self-coordination, both of which have negative downstream effects for interaction outcomes (e.g., lower liking and worse interaction quality). Together, these findings indicate that the functions of nonverbal social cues and the subsequent judgments receivers make are strongly affected by the presence of social partners and the interaction environment. Thus, because nonverbal communication contingencies change as a function of individuals, situations, and interaction modalities, investigations of nonverbal cues should prioritize diverse social contexts to foster a well-rounded understanding of nonverbal behaviour

Interaction analytics for automatic assessment of communication quality in primary care

Effective doctor-patient communication is a crucial element of health care, influencing patients’ personal and medical outcomes following the interview. The set of skills used in interpersonal interaction is complex, involving verbal and non-verbal behaviour. Precise attributes of good non-verbal behaviour are difficult to characterise, but models and studies offer insight on relevant factors. In this PhD, I studied how the attributes of non-verbal behaviour can be automatically extracted and assessed, focusing on turn-taking patterns of and the prosody of patient-clinician dialogues. I described clinician-patient communication and the tools and methods used to train and assess communication during the consultation. I then proceeded to a review of the literature on the existing efforts to automate assessment, depicting an emerging domain focused on the semantic content of the exchange and a lack of investigation on interaction dynamics, notably on the structure of turns and prosody. To undertake the study of these aspects, I initially planned the collection of data. I underlined the need for a system that follows the requirements of sensitive data collection regarding data quality and security. I went on to design a secure system which records participants’ speech as well as the body posture of the clinician. I provided an open-source implementation and I supported its use by the scientific community. I investigated the automatic extraction and analysis of some non-verbal components of the clinician-patient communication on an existing corpus of GP consultations. I outlined different patterns in the clinician-patient interaction and I further developed explanations of known consulting behaviours, such as the general imbalance of the doctor-patient interaction and differences in the control of the conversation. I compared behaviours present in face to face, telephone, and video consultations, finding overall similarities alongside noticeable differences in patterns of overlapping speech and switching behaviour. I further studied non-verbal signals by analysing speech prosodic features, investigating differences in participants’ behaviour and relations between the assessment of the clinician-patient communication and prosodic features. While limited in their interpretative power on the explored dataset, these signals nonetheless provide additional metrics to identify and characterise variations in the non-verbal behaviour of the participants. Analysing clinician-patient communication is difficult even for human experts. Automating that process in this work has been particularly challenging. I demonstrated the capacity of automated processing of non-verbal behaviours to analyse clinician-patient communication. I outlined the ability to explore new aspects, interaction dynamics, and objectively describe how patients and clinicians interact. I further explained known aspects such as clinician dominance in more detail. I also provided a methodology to characterise participants’ turns taking behaviour and speech prosody for the objective appraisal of the quality of non-verbal communication. This methodology is aimed at further use in research and education