Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour

Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.Comment: 12 pages, 6 figure

Dissociation and interpersonal autonomic physiology in psychotherapy research: an integrative view encompassing psychodynamic and neuroscience theoretical frameworks

Interpersonal autonomic physiology is an interdisciplinary research field, assessing the relational interdependence of two (or more) interacting individual both at the behavioral and psychophysiological levels. Despite its quite long tradition, only eight studies since 1955 have focused on the interaction of psychotherapy dyads, and none of them have focused on the shared processual level, assessing dynamic phenomena such as dissociation. We longitudinally observed two brief psychodynamic psychotherapies, entirely audio and video-recorded (16 sessions, weekly frequency, 45 min.). Autonomic nervous system measures were continuously collected during each session. Personality, empathy, dissociative features and clinical progress measures were collected prior and post therapy, and after each clinical session. Two-independent judges, trained psychotherapist, codified the interactions\u2019 micro-processes. Time-series based analyses were performed to assess interpersonal synchronization and de-synchronization in patient\u2019s and therapist\u2019s physiological activity. Psychophysiological synchrony revealed a clear association with empathic attunement, while desynchronization phases (range of length 30-150 sec.) showed a linkage with dissociative processes, usually associated to the patient\u2019s narrative core relational trauma. Our findings are discussed under the perspective of psychodynamic models of Stern (\u201cpresent moment\u201d), Sander, Beebe and Lachmann (dyad system model of interaction), Lanius (Trauma model), and the neuroscientific frameworks proposed by Thayer (neurovisceral integration model), and Porges (polyvagal theory). The collected data allows to attempt an integration of these theoretical approaches under the light of Complex Dynamic Systems. The rich theoretical work and the encouraging clinical results might represents a new fascinating frontier of research in psychotherapy

Nägemistaju automaatsete protsesside eksperimentaalne uurimine

Väitekirja elektrooniline versioon ei sisalda publikatsiooneVäitekiri keskendub nägemistaju protsesside eksperimentaalsele uurimisele, mis on suuremal või vähemal määral automaatsed. Uurimistöös on kasutatud erinevaid eksperimentaalseid katseparadigmasid ja katsestiimuleid ning nii käitumuslikke- kui ka ajukuvamismeetodeid. Esimesed kolm empiirilist uurimust käsitlevad liikumisinformatsiooni töötlust, mis on evolutsiooni käigus kujunenud üheks olulisemaks baasprotsessiks nägemistajus. Esmalt huvitas meid, kuidas avastatakse liikuva objekti suunamuutusi, kui samal ajal toimub ka taustal liikumine (Uurimus I). Nägemistaju uurijad on pikka aega arvanud, et liikumist arvutatakse alati mõne välise objekti või tausta suhtes. Meie uurimistulemused ei kinnitanud taolise suhtelise liikumise printsiibi paikapidavust ning toetavad pigem seisukohta, et eesmärkobjekti liikumisinformatsiooni töötlus on automaatne protsess, mis tuvastab silma põhjas toimuvaid nihkeid, ja taustal toimuv seda eriti ei mõjuta. Teise uurimuse tulemused (Uurimus II) näitasid, et nägemissüsteem töötleb väga edukalt ka seda liikumisinformatsiooni, millele vaatleja teadlikult tähelepanu ei pööra. See tähendab, et samal ajal, kui inimene on mõne tähelepanu hõlmava tegevusega ametis, suudab tema aju taustal toimuvaid sündmusi automaatselt registreerida. Igapäevaselt on inimese nägemisväljas alati palju erinevaid objekte, millel on erinevad omadused, mistõttu järgmiseks huvitas meid (Uurimus III), kuidas ühe tunnuse (antud juhul värvimuutuse) töötlemist mõjutab mõne teise tunnusega toimuv (antud juhul liikumiskiiruse) muutus. Näitasime, et objekti liikumine parandas sama objekti värvimuutuse avastamist, mis viitab, et nende kahe omaduse töötlemine ajus ei ole päris eraldiseisev protsess. Samuti tähendab taoline tulemus, et hoolimata ühele tunnusele keskendumisest ei suuda inimene ignoreerida teist tähelepanu tõmbavat tunnust (liikumine), mis viitab taas kord automaatsetele töötlusprotsessidele. Neljas uurimus keskendus emotsionaalsete näoväljenduste töötlusele, kuna need kannavad keskkonnas hakkamasaamiseks vajalikke sotsiaalseid signaale, mistõttu on alust arvata, et nende töötlus on kujunenud suuresti automaatseks protsessiks. Näitasime, et emotsiooni väljendavaid nägusid avastati kiiremini ja kergemini kui neutraalse ilmega nägusid ning et vihane nägu tõmbas rohkem tähelepanu kui rõõmus (Uurimus IV). Väitekirja viimane osa puudutab visuaalset lahknevusnegatiivsust (ingl Visual Mismatch Negativity ehk vMMN), mis näitab aju võimet avastada automaatselt erinevusi enda loodud mudelist ümbritseva keskkonna kohta. Selle automaatse erinevuse avastamise mehhanismi uurimisse andsid oma panuse nii Uurimus II kui Uurimus IV, mis mõlemad pakuvad välja tõendusi vMMN tekkimise kohta eri tingimustel ja katseparadigmades ning ka vajalikke metodoloogilisi täiendusi. Uurimus V on esimene kogu siiani ilmunud temaatilist teadustööd hõlmav ülevaateartikkel ja metaanalüüs visuaalsest lahknevusnegatiivsusest psühhiaatriliste ja neuroloogiliste haiguste korral, mis panustab oluliselt visuaalse lahknevusnegatiivsuse valdkonna arengusse.The research presented and discussed in the thesis is an experimental exploration of processes in visual perception, which all display a considerable amount of automaticity. These processes are targeted from different angles using different experimental paradigms and stimuli, and by measuring both behavioural and brain responses. In the first three empirical studies, the focus is on motion detection that is regarded one of the most basic processes shaped by evolution. Study I investigated how motion information of an object is processed in the presence of background motion. Although it is widely believed that no motion can be perceived without establishing a frame of reference with other objects or motion on the background, our results found no support for relative motion principle. This finding speaks in favour of a simple and automatic process of detecting motion, which is largely insensitive to the surrounding context. Study II shows that the visual system is built to automatically process motion information that is outside of our attentional focus. This means that even if we are concentrating on some task, our brain constantly monitors the surrounding environment. Study III addressed the question of what happens when multiple stimulus qualities (motion and colour) are present and varied, which is the everyday reality of our visual input. We showed that velocity facilitated the detection of colour changes, which suggests that processing motion and colour is not entirely isolated. These results also indicate that it is hard to ignore motion information, and processing it is rather automatically initiated. The fourth empirical study focusses on another example of visual input that is processed in a rather automatic way and carries high survival value – emotional expressions. In Study IV, participants detected emotional facial expressions faster and more easily compared with neutral facial expressions, with a tendency towards more automatic attention to angry faces. In addition, we investigated the emergence of visual mismatch negativity (vMMN) that is one of the most objective and efficient methods for analysing automatic processes in the brain. Study II and Study IV proposed several methodological gains for registering this automatic change-detection mechanism. Study V is an important contribution to the vMMN research field as it is the first comprehensive review and meta-analysis of the vMMN studies in psychiatric and neurological disorders

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

A New Perspective on Assessing Cognition in Children through Estimating Shared Intentionality

This theoretical article aims to create a conceptual framework for future research on digital methods for assessing cognition in children through estimating shared intentionality, different from assessing through behavioral markers. It shows the new assessing paradigm based directly on the evaluation of parent-child interaction exchanges (protoconversation), allowing early monitoring of children’s developmental trajectories. This literature analysis attempts to understand how cognition is related to emotions in interpersonal dynamics and whether assessing these dynamics shows cognitive abilities in children. The first part discusses infants’ unexpected achievements, observing the literature about children’s development. The analysis supposes that due to the caregiver’s help under emotional arousal, newborns’ intentionality could appear even before it is possible for children’s intention to occur. The emotional bond evokes intentionality in neonates. Therefore, they can manifest unexpected achievements while performing them with caregivers. This outcome shows an appearance of protoconversation in adult-children dyads through shared intentionality. The article presents experimental data of other studies that extend our knowledge about human cognition by showing an increase of coordinated neuronal activities and the acquisition of new knowledge by subjects in the absence of sensory cues. This highlights the contribution of interpersonal interaction to gain cognition, discussed already by Vygotsky. The current theoretical study hypothesizes that if shared intentionality promotes cognition from the onset, this interaction modality can also facilitate cognition in older children. Therefore in the second step, the current article analyzes empirical data of recent studies that reported meaningful interaction in mother-infant dyads without sensory cues. It discusses whether an unbiased digital assessment of the interaction ability of children is possible before the age when the typical developmental trajectory implies verbal communication. The article develops knowledge for a digital assessment that can measure the extent of children’s ability to acquire knowledge through protoconversation. This specific assessment can signalize the lack of communication ability in children even when the typical trajectory of peers’ development does not imply verbal communication.publishersversionPeer reviewe

Fusing face and body gesture for machine recognition of emotions

Research shows that humans are more likely to consider computers to be human-like when those computers understand and display appropriate nonverbal communicative behavior. Most of the existing systems attempting to analyze the human nonverbal behavior focus only on the face; research that aims to integrate gesture as an expression mean has only recently emerged. This paper presents an approach to automatic visual recognition of expressive face and upper body action units (FAUs and BAUs) suitable for use in a vision-based affective multimodal framework. After describing the feature extraction techniques, classification results from three subjects are presented. Firstly, individual classifiers are trained separately with face and body features for classification into FAU and BAU categories. Secondly, the same procedure is applied for classification into labeled emotion categories. Finally, we fuse face and body information for classification into combined emotion categories. In our experiments, the emotion classification using the two modalities achieved a better recognition accuracy outperforming the classification using the individual face modality. © 2005 IEEE