Analysing the Direction of Emotional Influence in Nonverbal Dyadic Communication: A Facial-Expression Study

Identifying the direction of emotional influence in a dyadic dialogue is of increasing interest in the psychological sciences with applications in psychotherapy, analysis of political interactions, or interpersonal conflict behavior. Facial expressions are widely described as being automatic and thus hard to overtly influence. As such, they are a perfect measure for a better understanding of unintentional behavior cues about social-emotional cognitive processes. With this view, this study is concerned with the analysis of the direction of emotional influence in dyadic dialogue based on facial expressions only. We exploit computer vision capabilities along with causal inference theory for quantitative verification of hypotheses on the direction of emotional influence, i.e., causal effect relationships, in dyadic dialogues. We address two main issues. First, in a dyadic dialogue, emotional influence occurs over transient time intervals and with intensity and direction that are variant over time. To this end, we propose a relevant interval selection approach that we use prior to causal inference to identify those transient intervals where causal inference should be applied. Second, we propose to use fine-grained facial expressions that are present when strong distinct facial emotions are not visible. To specify the direction of influence, we apply the concept of Granger causality to the time series of facial expressions over selected relevant intervals. We tested our approach on newly, experimentally obtained data. Based on the quantitative verification of hypotheses on the direction of emotional influence, we were able to show that the proposed approach is most promising to reveal the causal effect pattern in various instructed interaction conditions.Comment: arXiv admin note: text overlap with arXiv:1810.1217

Finding Structure in Time:Visualizing and Analyzing Behavioral Time Series

The temporal structure of behavior contains a rich source of information about its dynamic organization, origins, and development. Today, advances in sensing and data storage allow researchers to collect multiple dimensions of behavioral data at a fine temporal scale both in and out of the laboratory, leading to the curation of massive multimodal corpora of behavior. However, along with these new opportunities come new challenges. Theories are often underspecified as to the exact nature of these unfolding interactions, and psychologists have limited ready-to-use methods and training for quantifying structures and patterns in behavioral time series. In this paper, we will introduce four techniques to interpret and analyze high-density multi-modal behavior data, namely, to: (1) visualize the raw time series, (2) describe the overall distributional structure of temporal events (Burstiness calculation), (3) characterize the non-linear dynamics over multiple timescales with Chromatic and Anisotropic Cross-Recurrence Quantification Analysis (CRQA), (4) and quantify the directional relations among a set of interdependent multimodal behavioral variables with Granger Causality. Each technique is introduced in a module with conceptual background, sample data drawn from empirical studies and ready-to-use Matlab scripts. The code modules showcase each technique's application with detailed documentation to allow more advanced users to adapt them to their own datasets. Additionally, to make our modules more accessible to beginner programmers, we provide a "Programming Basics" module that introduces common functions for working with behavioral timeseries data in Matlab. Together, the materials provide a practical introduction to a range of analyses that psychologists can use to discover temporal structure in high-density behavioral data.</p

Neural and motor basis of inter-individual interactions

The goal of my Ph.D. work was to investigate the behavioral markers and the brain activities responsible for the emergence of sensorimotor communication. Sensorimotor communication can be defined as a form of communication consisting into flexible exchanges based on bodily signals, in order to increase the efficiency of the inter-individual coordination. For instance, a soccer player carving his movements to inform another player about his intention. This form of interaction is highly dependent of the motor system and the ability to produce appropriate movements but also of the ability of the partner to decode these cues. To tackle these facets of human social interaction, we approached the complexity of the problem by splitting my research activities into two separate lines of research. First, we pursued the examination of motor-based humans\u2019 capability to perceive and \u201cread\u201d other\u2019s behaviors in focusing on single-subject experiment. The discovery of mirror neurons in monkey premotor cortex in the early nineties (di Pellegrino et al. 1992) motivated a number of human studies on this topic (Rizzolatti and Craighero 2004). The critical finding was that some ventral premotor neurons are engaged during visual presentation of actions performed by conspecifics. More importantly, those neurons were shown to encode also the actual execution of similar actions (i.e. irrespective of who the acting individual is). This phenomenon has been highly investigated in humans by using cortical and cortico-spinal measures (for review see, fMRI: Molenberghs, Cunnington, and Mattingley 2012; TMS: Naish et al. 2014; EEG: Pineda 2008). During single pulse TMS (over the primary motor cortex), the amplitude of motor evoked potentials (MEPs) provides an index of corticospinal recruitment. During action observation the modulation of this index follow the expected changes during action execution (Fadiga et al. 1995). However, dozens of studies have been published on this topic and revealed important inconsistencies. For instance, MEPs has been shown to be dependent on observed low-level motor features (e.g. kinematic features or electromyography temporal coupling; Gangitano, Mottaghy, and Pascual-Leone 2001; Borroni et al. 2005; Cavallo et al. 2012) as well as high level movement properties (e.g. action goals; Cattaneo et al. 2009; Cattaneo et al. 2013). Furthermore, MEPs modulations do not seem to be related to the observed effectors (Borroni and Baldissera 2008; Finisguerra et al. 2015; Senna, Bolognini, and Maravita 2014), suggesting their independence from low-level movement features. These contradictions call for new paradigms. Our starting hypothesis here is that the organization and function of the mirror mechanism should follow that of the motor system during action execution. Hence, we derived three action observation protocols from classical motor control theories: 1) The first study was motivated by the fact that motor redundancy in action execution do not allow the presence of a one-to-one mapping between (single) muscle activation and action goals. Based on that, we showed that the effect of action observation (observation of an actor performing a power versus a precision grasp) are variable at the single muscle level (MEPs; motor evoked potentials) but robust when evaluating the kinematic of TMS-evoked movements. Considering that movements are based on the coordination of multiple muscle activations (muscular synergies), MEPs may represent a partial picture of the real corticospinal activation. Inversely, movement kinematics is both the final functional byproduct of muscles coordination and the sole visual feedback that can be extracted from action observation (i.e. muscle recruitment is not visible). We conclude that TMS-evoked kinematics may be more reliable in representing the state of the motor system during action observation. 2) In the second study, we exploited the inter-subject variability inherent to everyday whole-body human actions, to evaluate the link between individual motor signatures (or motor styles) and other\u2019s action perception. We showed no group-level effect but a robust correlation between the individual motor signature recorded during action execution and the subsequent modulations of corticospinal excitability during action observation. However, results were at odds with a strict version of the direct matching hypothesis that would suggest the opposite pattern. In fact, the more the actor\u2019s movement was similar to the observer\u2019s individual motor signature, the smaller was the MEPs amplitude, and vice versa. These results conform to the predictive coding hypothesis, suggesting that during AO, the motor system compares our own way of doing the action (individual motor signature) with the action displayed on the screen (actor\u2019s movement). 3) In the third study, we investigated the neural mechanisms underlying the visual perception of action mistakes. According to a strict version of the direct matching hypothesis, the observer should potentially reproduce the neural activation present during the actual execution of action errors (van Schie et al. 2004). Here, instead of observing an increase of cortical inhibition, we showed an early (120 ms) decrease of intracortical inhibition (short intracortical inhibition) when a mismatch was present between the observed action (erroneous) and the observer\u2019s expectation. As proposed by the predictive coding framework, the motor system may be involved in the generation of an error signal potentially relying on an early decrease of intracortical inhibition within the corticomotor system. The second line of research aimed at the investigation of how sensorimotor communication flows between agents engaged in a complementary action coordination task. In this regard, measures of interest where related to muscle activity and/or kinematics as the recording of TMS-related indexes would be too complicated in a joint-action scenario. 1) In the first study, we exploited the known phenomenon of Anticipatory Postural Adjustments (APAs). APAs refers to postural adjustments made in anticipation of a self- or externally-generated disturbance in order to cope for the predicted perturbation and stabilize the current posture. Here we examined how observing someone else lifting an object we hold can affect our own anticipatory postural adjustments of the arm. We showed that the visual information alone (joint action condition), in the absence of efference copy (present only when the subject is unloading by himself the object situated on his hand), were not sufficient to fully deploy the needed anticipatory muscular activations. Rather, action observation elicited a dampened APA response that is later augmented by the arrival of tactile congruent feedback. 2) In a second study, we recorded the kinematic of orchestra musicians (one conductor and two lines of violinists). A manipulation was added to perturb the normal flow of information conveyed by the visual channel. The first line of violinist where rotated 180\ub0, and thus faced the second line. Several techniques were used to extract inter-group (Granger Causality method) and intra-group synchronization (PCA for musicians and autoregression for conductors). The analyses were directed to two kinematic features, hand and head movements, which are central for functionally different action. The hand is essential for instrumental actions, whereas head movements encode ancillary expressive actions. During the perturbation, we observed a complete reshaping of the whole patterns of communication going in the direction of a distribution of the leadership between conductor and violinists, especially for what regards head movements. In fact, in the perturbed condition, the second line acts as an informational hub connecting the first line to the conductor they no longer can see. This study evidences different forms of communications (coordination versus synchronization) flowing via different channels (ancillary versus instrumental) with different time-scales

Brains in interaction

Wanneer twee mensen met elkaar communiceren, dan ontstaat er een soort tijdelijke verbinding tussen hen. Deze verbinding bestaat uit een keten van gebeurtenissen en begint bijvoorbeeld bij de hersenactiviteit in de motorische cortex van de ene persoon. Deze activiteit leidt tot gedrag, bijvoorbeeld het maken van een gebaar, dat wordt gezien door de andere persoon. Deze bekijkt en interpreteert dit gebaar wat leidt tot activiteit in de visuele, sensorische en associatieve cortices. Dit kan dan weer leiden tot hersenactiviteit in de motorische cortex, tot een ander gebaar en zo verder. Dit proefschrift beschrijft een hersenonderzoek naar zo’n dergelijke indirecte verbinding tussen twee mensen. Voordat ik in ga op het onderzoek wat we hebben uitgevoerd, is het belangrijk om iets te weten over de achtergrond en inspiratie waarop dit onderzoek is gebaseerd. De afgelopen jaren zijn er twee belangrijke ideeën ontwikkelt over hoe mensen elkaar begrijpen en met elkaar kunnen communiceren: het idee van een spiegelsysteem en een ‘redeneersysteem’. Deze twee ideeën vormen de basis van dit onderzoek en worden beschreven in de volgende paragraven. Verder heeft de ontwikkeling van ‘Granger causaliteit’, een analysemethode om verbindingen tussen hersengebieden vast te stellen ook een belangrijke rol gespeeld, deze wordt hierna beschreven. HE T S P I EGE L S Y S T E EM Het idee van spiegelen is dat ons brein de handelingen van andere mensen ‘nabootst’. Aan de basis van dit idee staat de ontdekking van spiegelneuronen (‘mirror neurons’) in de jaren negentig (Gallese et al., 1996; Pellegrino et al., 1992). Deze spiegelneuronen zijn min of meer per toeval ontdekt in een lab in Parma tijdens het meten van neuronen in het gebied F5 (ventrale premotorische cortex) van deMakaak aap. Een onderzoeker merkte op dat deze neuronen niet alleen reageerden op het moment dat de aap zelf een pinda oppakte, maar ook op het moment dat de aap naar de onderzoeker keek terwijl deze een pinda oppakte. Het was bekend dat deze neuronen betrokken zijn bij het uitvoeren van doelgerichte handelingen met de handen en met de mond. Maar nu werd opeens duidelijk dat deze gebieden ook sensorische eigenschappen bezitten (Kurata and Tanji, 1986; Rizzolatti et al., 1988). Deze neuronen representeren hiermee zowel het uitvoeren van een handeling als de waarneming van die handeling. De ontdekking van spiegelneuronen had een grote impact, omdat hiermee het vermoeden werd bevestigd dat waarnemen en handelen sterk aan elkaar gekoppeld zijn. Dit idee speelde al langer een rol in psychologische theorieën. James Gibson beweerde bijvoorbeeld dat perceptie bestaat uit het direct waarnemen van handelingsmogelijkheden (Gibson, 1986). Kort na de eerste ontdekking van spiegelneuronen wilde men weten of de menselijke hersenen ook zo’n dergelijk mechanisme bezitten. Omdat het meten van een individuele neuron vrijwel niet mogelijk is zonder een brein te beschadigen, richtten onderzoeken zich op de vraag of er wellicht hersengebieden bestaan die activiteit laten zien tijdens zowel het uitvoeren als het waarnemen van een handeling (Buccino et al., 2001; Grafton et al., 1996; Grèzes et al., 1998; Grèzes and Decety, 2001; Grèzes et al., 2003; Nishitani and Hari, 2000, 2002; Perani et al., 2001; Gazzola et al., 2007b,a; Gazzola and Keysers, 2008). Dat blijkt inderdaad zo te zijn en de gebieden met deze eigenschap vormen samen het menselijke spiegelsysteem (Keysers and Gazzola, 2009). Deze gebieden zijn de ventrale en dorsale premotorische cortex, de inferieure parietale cortex en de middelste superieure temporele gyrus (zie Figuur 3). Er bestaan overigens niet alleen spiegelgebieden die een overlap in activiteit laten zien voor het uitvoeren en waarnemen van handelingen, maar ook voor het ervaren en het waarnemen van emoties en sensaties, zoals walging, aanraking en pijn (Wicker et al., 2003; Keysers and Perrett, 2004; Singer et al., 2004; Bastiaansen et al., 2009). Innovatieve experimenten, die bijvoorbeeld gebruikmaken van ‘cross-modal repetition suppression’, hebben inmiddels wetenschappelijk bewijs geleverd voor het bestaan van individuele spiegelneuronen in de menselijke hersenen (Kilner et al., 2009; Lingnau et al., 2009; Chong et al., 2008;Mukamel et al., 2010). Doordat spiegelneuronen een directe link leggen tussen de handelingen die we zelf uitvoeren en de handelingen die we anderen zien doen, wordt aangenomen dat spiegelneuronen een functie hebben in het begrijpen van wat de ander aan het doen is (zie Rizzolatti and Sinigaglia, 2010, voor een recent overzicht van de literatuur). Bij het zien van een handeling van iemand anders wordt de motorische representatie van deze handeling in de eigen hersenen actief, alsof deze handeling zelf wordt uitgevoerd. Dit idee vormt de kern van de simulatietheorie: we begrijpen wat een ander doet doordat we deze handeling als het ware simuleren in onze eigen hersenen (Goldman, 1992; Gibson, 1986; Gallese, 2003). Belangrijk voor het onderzoek in dit proefschrift is dat de simulatietheorie een voorspellingmaakt over spiegelneuronen. Deze theorie beweert namelijk dat spiegelneuronen in het brein van degene die een handeling waarneemt resoneren met de spiegelneuronen van degene die de handeling uitvoert. De term‘resonantie’ wordt hier losjes gebruikt en er wordt mee bedoeld dat de pieken en dalen in de hersenactiviteit van het motorsysteem van de ene persoon overeenkomstige pieken en dalen veroorzaakt in de hersenactiviteit van het motorsysteem in de andere persoon (Gallese and Goldman, 1998; Gallese et al., 2004; Rizzolatti et al., 2001). In Hoofdstuk 4 van dit proefschrift wordt deze bewering over resonantie onderzocht. HE T REDENE ERS Y S T E EM Naast dit spiegelmechanisme waarmee we anderen begrijpen, bezitten we ook een meer reflectief vermogen om na te denken over wat er in anderen omgaat. Denk bijvoorbeeld aan een typische scene uit een soap, zoals The Bold and the Beautiful: Taylor and Ridge staan op het punt om met elkaar in het huwelijk te treden. Zonder dat Taylor dit weet, staat Brooke op het punt om te vertellen dat ze zwanger is van Ridge, hopende dat ze hiermee de bruiloft kan verhinderen. Om zo’n situatie te kunnen begrijpen en te kunnen waarderen, moeten we in staat zijn om bij te houden wat de verschillende personen wel en niet weten en wat ze zullen denken op het moment dat ze het te horen zullen krijgen. Dit soort bewuste denkprocessen wordt in de literatuur wel ‘Theory of Mind’ (ToM) genoemd (Premack andWoodruff, 1978; Wimmer and Perner, 1983) en vindt plaats in andere gebieden dan de spiegelgebieden (Frith and Frith, 1999, 2006). Het zijn de ‘redeneergebieden’ (zie Figuur 3), die actief zijn tijdens bijvoorbeeld het interpreteren van (strip)verhalen en het nadenken over jezelf en anderen (Amodio and Frith, 2006; Fletcher et al., 1995; Frith and Frith, 2006, 2003; Gallagher et al., 2000; Gusnard et al., 2001). De belangrijkste twee gebieden van dit redeneersysteem zijn de ventrale mediale prefrontale cortex en de temporeelparietale junctie. Over the decades, two important networks in the brain have been identified about how people interact: the mirror system and the mentalizing network. This thesis investigates how these networks work together during social interaction. We performed an experiment in which brain activity of two persons was measured while they engaged in a social communication game (Charades). Results showed that the mirror system is highly involved during the game, while the main mentalizing area does not show any involvement. We then extended a connectivity analysis, Granger causality, which is usually applied within one brain, to a between-brain analysis. With this method, we used brain activity of the gesturer to map regions in the brain of the guesser, whose brain activity has a Granger-causal relation to that of the gesturer. The mirror system of the gesturer shows a Granger-causal relation to the mirror system of the guesser, but also to the main mentalizing area of the guesser. This means that, even while this mentalizing area does not show involvement when analyzed using a classic method, it does show a temporal relationship with the brain activity of the gesturer. We furthermore performed simulations to investigate a possible confound of Granger causality: inter- and intrasubject variability in hemodynamic responses. Results show high sensitivity and accuracy for Granger causality between-brains, while sensitivity of within-brain Granger causality remains low. However, if a Grangercausality is found, this indicates the correct underlying direction in 80% of the cases. Finally, we used within-brain Granger causality to investigate how areas in the mirror system influence each other during gesturing and guessing.

Complexity Science in Human Change

This reprint encompasses fourteen contributions that offer avenues towards a better understanding of complex systems in human behavior. The phenomena studied here are generally pattern formation processes that originate in social interaction and psychotherapy. Several accounts are also given of the coordination in body movements and in physiological, neuronal and linguistic processes. A common denominator of such pattern formation is that complexity and entropy of the respective systems become reduced spontaneously, which is the hallmark of self-organization. The various methodological approaches of how to model such processes are presented in some detail. Results from the various methods are systematically compared and discussed. Among these approaches are algorithms for the quantification of synchrony by cross-correlational statistics, surrogate control procedures, recurrence mapping and network models.This volume offers an informative and sophisticated resource for scholars of human change, and as well for students at advanced levels, from graduate to post-doctoral. The reprint is multidisciplinary in nature, binding together the fields of medicine, psychology, physics, and neuroscience

An information-theoretic account of human–computer interaction

This thesis presents a theoretical framework for the study of interactive systems, using methods from information theory, machine learning and control theory. The framework builds on the information-theoretic capacities of empowerment, relevant information and mutual information, which I adapt and apply to the domain of human-computer interaction. Three user studies exploring dynamic interactive scenarios - one car-tracking and two collaborative target-acquisition experiments - provide empirical data for the development of probabilistic models, used in the characterisation of specific aspects of human performance, such as the level of control, the quality of decision-making, and the level of engagement in interpersonal coordination. Human control models are extended to accommodate for the inherent lags, characteristic for human-computer and human-human interaction, in a principled way. Optimal controllers, describing particular patterns of human behaviour, are built on these theoretical models, providing evidence for specific limits of human performance through simulations. The thesis describes the potential of empowerment, as a generic task-independent measure of control, to characterise the uncertainty in human-machine interfaces. This work builds an important bridge between theory and experiments, and suggests that the proposed information-theoretic concepts could provide analytical tools for supporting the design and evaluation of interactive systems, by elucidating novel aspects of human performance complementing standard measures. The thesis provides proof of concept examples for the application of such information-theoretic measures, and demonstrates how they can be treated naturally side-by-side along traditional metrics used in HCI research. It emphasises the acquisition cost of accurate theoretical models, necessary to ensure the reliability of such measures