Perception of Facial Emotional Expressions and Touch in Virtual Face-to-Face Interaction

Emotional expressions as manifested in facial movements, voice, and touch are a crucial part of face-to-face interaction. The majority of existing neuroscientific research on emotional expressions concerns the perception of unimodal emotional cues, such as facial emotional expressions. In natural face-to-face interaction, however, emotions are often expressed as compounds of facial, tactile, prosodic, and postural cues. How the brain processes such multimodal emotional information remains poorly understood. The aim of the current dissertation is to investigate how emotional expressions conveyed consecutively via face and touch are integrated in perceptual processing and decision-making. Four studies were conducted to measure event-related brain potentials (ERPs) and autonomic nervous system responses to simulated touches and facial emotional expression stimuli. The first two studies used virtual reality to investigate how a virtual agent’s facial emotional expressions influenced the way the agent’s subsequent touch was perceived (Study I) and whether the receiver’s individual characteristics influenced this visuo-tactile affective modulation (Study II). Touch perception was measured using self-reports, somatosensory-evoked potentials (SEPs), and cardiac orienting responses (ORs), and the individual characteristics were indexed by behavioural inhibition system sensitivity (BIS) and gender. Study III investigated whether receiving a touch influenced the processing of a subsequent emotional face picture presented on the computer screen. Here, face-evoked ERPs, ORs, and facial electromyography were measured. Finally, the Study IV examined whether a virtual agent’s touch and emotional facial expressions influence receivers’ decision-making and offer-related ORs in an economic decision-making game. Additionally, the study examined whether the receivers’ behavioural inhibition/approach system (BAS/BIS) sensitivities and sensitivity to unfair treatment moderated persuasiveness of nonverbal cues. Study I revealed that happy, angry, and sad facial expressions resulted in amplified SEPs around 20–50 ms after touch onset, whereas in later latencies (250–650 ms), the angry facial expression amplified and the happy expression decreased the SEP amplitudes. In Study II, men with high BIS were found to perceive touch from male agents as especially intense if accompanied by happy, angry, of fearful facial expressions, and they showed pronounced cardiac OR to all the touches. Study III demonstrated that receiving a computer-generated touch did not modulate emotional face processing in any of the measured indices. Finally, in Study IV, people were found to accept unfair offers more often if the agent smiled or touched them before making the offer. The touch had a stronger persuasive influence in people with low sensitivity to unfairness and low approach tendency, whereas the effect of facial expressions was moderated by BIS. Altogether, the findings of the dissertation reveal that a sender’s facial emotional expressions modulate subsequent touch perception at a very early stage and that the modulation is based on different emotional information in different temporal stages. In addition, the findings suggest that motivational tendencies and gender influence the manner in which people perceive a sender’s multimodal emotional expressions and make decisions thereafter. These findings are valuable for basic research, but their implications extend also to the development of novel clinical interventions and social virtual reality applications.Kasvonilmeiden, asentojen, äänenpainojen ja kosketuksen kautta välittyvät tunneilmaisut ovat keskeinen osa sosiaalista vuorovaikutusta. Suuri osa olemassa olevasta tunneilmaisujen neurotieteellisestä tutkimuksesta koskee unimodaalisten tunneilmaisuärsykkeiden, kuten kasvonilmeiden prosessointia. Kasvokkaisessa vuorovaikutuksessa tunneilmaisut esiintyvät kuitenkin usein kasvonilmeiden, äänenpainojen, asentojen ja kosketuksen sarjamaisina yhdistelminä. Aivojen tavasta prosessoida tällaisia moniaistisia tunneilmaisuja tiedetään vielä vähän. Tässä väitöskirjatutkimuksessa tutkittiin, miten peräkkäin esitetyt kasvonilmeet ja kosketus integroituvat toisiinsa aistiprosessoinnin tasolla ja vaikuttavat vastaanottajan sosiaaliseen päätöksentekoon. Väitöskirja koostuu neljästä tutkimuksesta, joissa mitattiin aivosähkökäyrän herätevasteita ja autonomisen hermoston reaktioita kosketukseen ja kasvonilmeisiin. Kahdessa ensimmäisessä tutkimuksessa selvitettiin, kuinka virtuaalitodellisuuteen heijastetun ihmishahmon kasvonilmeet (iloinen, suuttunut, pelokas, surullinen ja neutraali) vaikuttavat hahmon lähettämän koneellisesti tuotetun kosketuksen havaitsemiseen (tutkimus I) ja kuinka vastaanottajan välttämistaipumus ja sukupuoli säätelevät kasvonilmeiden vaikutuksia kosketuksen havaitsemiseen (tutkimus II). Kosketuksen havaitsemista mitattiin itseraportoinnin, somatosensoristen herätevasteiden ja sydämen orientaatiorefleksin avulla. Tutkimuksessa III selvitettiin, vaikuttaako koneellisesti tuotettu kosketus sitä seuraavan emotionaalisen kasvokuvan prosessointiin. Kasvonilmeiden prosessointia mitattiin kasvoärsykkeiden aikaansaamien näköherätevasteiden, kasvolihasaktivaation ja sydämen orientaatiorefleksin avulla. Tutkimuksessa IV tarkasteltiin virtuaalihahmon kasvonilmeiden ja kosketuksen yhteisvaikutuksia hahmon taloudellisiin tarjouksiin suostumiseen ja tarjousten aiheuttamaan sydämen orientaatiorefleksiin. Samalla tarkasteltiin, mikäli vastaanottajan lähestymis- ja välttämistaipumus sekä herkkyys epäreilulle kohtelulle säätelevät tunneilmaisuiden suostuttelevaa vaikutusta. Tutkimuksen I tulokset osoittivat, että kosketukset, joita edelsi iloinen, suuttunut tai surullinen ilme johtivat voimakkaampiin varhaisiin herätevasteisiin 20 - 50 millisekuntia kosketuksen esittämisestä. Vastaavasti myöhemmät vasteet (250-650 ms) olivat korostuneempia, jos hahmolla oli suuttunut ilme, ja latistuneita jos hahmo hymyili. Tutkimuksessa II havaittiin, että erityisesti miehillä, joilla oli korkea välttämistaipumus, hahmon kosketus sai aikaan voimakkaan sydämen orientaatiorefleksin. He myös kokivat kosketuksen voimakkaana, kun hahmo ilmaisi kasvoillaan suuttumusta, pelkoa tai iloa koskettaessaan. Vastoin oletuksia, tutkimuksessa III koneellisesti tuotettu kosketus ei vaikuttanut kasvonilmeärsykkeen prosessointiin millään mitatuista vasteista. Tutkimuksessa IV havaittiin, että virtuaalihahmon kosketus ja iloinen/neutraali kasvonilme lisäsivät suostuvaisuutta hahmon esittämiin epäreiluihin taloudellisiin tarjouksiin. Kosketus johti suurempaan suostuvaisuuteen henkilöillä, joilla oli alhainen herkkyys epäreiluudelle ja alhainen lähestymistaipumus. Kasvonilmeiden vaikutusta suostuvaisuuteen sääteli vastaanottajan välttämistaipumus. Yhteenvetona tulokset osoittavat, että vuorovaikutuksessa ilmaistut kasvonilmeet ja kosketus integroituvat toisiinsa hyvin varhain ja että integraatioprosessi on monivaiheinen. Myös persoonallisuuspiirteillä ja sukupuolella näyttää olevan merkittävä rooli multimodaalisten tunneilmaisuiden havaitsemisessa ja vaikutuksessa päätöksentekoon

Anticipation of aversive visual stimuli lengthens perceived temporal duration

Subjective estimates of elapsed time are sensitive to the fluctuations in an emotional state. While it is well known that dangerous and threatening situations, such as electric shocks or loud noises, are perceived as lasting longer than safe events, it remains unclear whether anticipating a threatening event speeds up or slows down subjective time and what defines the direction of the distortion. We examined whether the anticipation of uncertain visual aversive events resulted in either underestimation or overestimation of perceived duration. The participants did a temporal bisection task, where they estimated durations of visual cues relative to previously learnt long and short standard durations. The colour of the to-be-timed visual cue signalled either a 50% or 0% probability of encountering an aversive image at the end of the interval. The cue durations were found to be overestimated due to anticipation of aversive images, even when no image was shown afterwards. Moreover, the overestimation was more pronounced in people who reported feeling more anxious while anticipating the image. These results demonstrate that anxiogenic anticipation of uncertain visual threats induce temporal overestimation, which questions a recently proposed view that temporal underestimation evoked by uncertain threats is due to anxiety.Peer reviewe

Time to imagine moving: Simulated motor activity affects time perception

Sensing the passage of time is important for countless daily tasks, yet time perception is easily influenced by perception, cognition, and emotion. Mechanistic accounts of time perception have traditionally regarded time perception as part of central cognition. Since proprioception, action execution, and sensorimotor contingencies also affect time perception, perception-action integration theories suggest motor processes are central to the experience of the passage of time. We investigated whether sensory information and motor activity may interactively affect the perception of the passage of time. Two prospective timing tasks involved timing a visual stimulus display conveying optical flow at increasing or decreasing velocity. While doing the timing tasks, participants were instructed to imagine themselves moving at increasing or decreasing speed, independently of the optical flow. In the direct-estimation task, the duration of the visual display was explicitly judged in seconds while in the motor-timing task, participants were asked to keep a constant pace of tapping. The direct-estimation task showed imagining accelerating movement resulted in relative overestimation of time, or time dilation, while decelerating movement elicited relative underestimation, or time compression. In the motor-timing task, imagined accelerating movement also accelerated tapping speed, replicating the time-dilation effect. The experiments show imagined movement affects time perception, suggesting a causal role of simulated motor activity. We argue that imagined movements and optical flow are integrated by temporal unfolding of sensorimotor contingencies. Consequently, as physical time is relative to spatial motion, so too is perception of time relative to imaginary motion.Peer reviewe

How bodily expressions of emotion after norm violation influence perceivers’ moral judgments and prevent social exclusion : A socio-functional approach to nonverbal shame display

According to a socio-functional perspective on emotions, displaying shame with averted gaze and a slumped posture following a norm violation signals that the person is ready to conform to the group’s moral standards, which in turn protects the person from social isolation and punishment. Although the assumption is intuitive, direct empirical evidence for it remains surprisingly limited and the mediating social-psychological mechanisms are poorly understood. Therefore, three experimental studies were conducted to investigate the social function of nonverbal displays of shame in the context of everyday norm violations. In Study 1, participants evaluated ten different expressions of emotion in regard to their affective valence, arousal, dominance, as well as social meaning in the context of norm violations. Displays of shame and sadness were seen as the most similar expressions with respect to the three affective dimensions and were perceived to communicate the perpetrator’s understanding of the group’s moral standards most effectively. In Study 2, participants read vignettes concerning norm violations and afterward saw a photograph of the perpetrator displaying nonverbal shame, sadness or a neutral expression. Perpetrators’ displays of shame and sadness increased perceived moral sense and amplified the observers’ willingness to cooperate with the perpetrators. However, neither display weakened the observer’s willingness to punish the perpetrator. In Study 3, the perpetrator was shown to display shame, sadness, anger or a neutral expression after getting caught at mild or severe norm violation. The results replicated previous findings but revealed also that the social effects of shame and sadness displays on punitive and cooperative intentions were mediated by different social appraisals. For example, display of shame uniquely reduced punitive intentions by increasing the perpetrator’s perceived moral sense, whereas expressions of both shame and sadness evoked empathy in the observers, which in turn reduced the punitive intentions. These results give support to the assumption that nonverbal shame displays serve a unique social function in preventing moral punishment and social exclusion. However, this support is only partial as the social functions of displaying shame are largely parallel to those of expressing sadness in the situation.Peer reviewe

Persuaded by the machine : The effect of virtual nonverbal cues and individual differences on compliance in economic bargaining

Receiving a touch or smile increases compliance in natural face-to-face settings. It has been unclear, however, whether a virtual agent’s touch and smile also promote compliance or whether there are individual differences in proneness to nonverbal persuasion. Utilising a multimodal virtual reality, we investigated whether touch and smile promoted compliance to a virtual agent’s requests and whether receiver’s personality modulated the effects. Compliance was measured using the ultimatum game, in which participants were asked to either reject or accept an agent’s monetary offers. Decision-making data were accompanied by offer-related cardiac responses, both of which were analyzed as a function of expression (anger, neutral, and happiness), touch (visuo-tactile, visual, no touch), and three personality traits: behavioral inhibition/activation system sensitivity (BIS/BAS) and justice sensitivity. People accepted unfair offers more often if the agents smiled or touched them. The effect of touch was more enhanced in those with low justice sensitivity and BAS, whereas facial expressions affected those with high BIS the most. Unfair offers amplified the cardiac response, but this effect was not dependent on nonverbal cues. Together, the results suggest that virtual nonverbal behaviors of virtual agents increase compliance and that there is substantial interindividual variation in proneness to persuasion.Receiving a touch or smile increases compliance in natural face-to-face settings. It has been unclear, however, whether a virtual agent's touch and smile also promote compliance or whether there are individual differences in proneness to nonverbal persuasion. Utilizing a multimodal virtual reality, we investigated whether touch and smile promoted compliance to a virtual agent's requests and whether receiver's personality modulated the effects. Compliance was measured using the ultimatum game, in which participants were asked to either reject or accept an agent's monetary offers. Decision-making data were accompanied by offer-related cardiac responses, both of which were analyzed as a function of expression (anger, neutral, and happiness), touch (visuo-tactile, visual, no touch), and three personality traits: behavioral inhibition/activation system sensitivity (BIS/BAS) and justice sensitivity. People accepted unfair offers more often if the agents smiled or touched them. The effect of touch was more enhanced in those with low justice sensitivity and BAS, whereas facial expressions affected those with high BIS the most. Unfair offers amplified the cardiac response, but this effect was not dependent on nonverbal cues. Together, the results suggest that virtual nonverbal behaviors of virtual agents increase compliance and that there is substantial interindividual variation in proneness to persuasion.Peer reviewe

Does Mediated Social Touch Succesfully Approximate Natural Social Touch?

Mediated social touch (MST) devices are upcoming. To date, experiments aimed at demonstrating whether effects of naturalistic social touch can be replicated with MST provide mixed findings. A possible explanation could be a lack of realism of current haptic displays in combination with not sufficiently taking contextual factors of social touch into account. Using a qualitative approach, our study aims to gain more insight into the influence of contextual effects on the experience of an MST, by means of exploring female participants’ experiences of receiving an MST from a male stranger versus their romantic partner. Our findings show that simultaneously feeling and seeing the touch act performed on a corporeal object can be beneficial for the MST experience. However, our findings also demonstrate that it is not self-evident to regard MST as phenomenologically equal to natural social touch, as it often fails to meet the expectations people have formed based on naturalistic social touch.To date, experiments aimed at demonstrating whether effects of naturalistic social touch can be replicated with Mediated Social Touch (MST) provide mixed findings. A possible explanation could be a lack of realism of current haptic displays in combination with not sufficiently taking contextual factors of social touch into account. Using a qualitative approach, our study aims to gain more insight into the influence of contextual effects on experience of an MST, by means of exploring female participants’ experiences of receiving an MST from a male stranger versus their romantic partner. Our findings show simultaneously feeling and seeing the touch act performed on a corporeal object can be beneficial for MST experience. However, our findings also demonstrate that it is not self-evident to regard MST as phenomenologically equal to natural social touch, as it often fails to meet expectations people formed based on naturalistic social touch.Peer reviewe

Touching virtual humans: Haptic responses reveal the emotional impact of affective agents

Interpersonal touch is critical for social-emotional development and presents a powerful modality for communicating emotions. Virtual agents of the future could capitalize on touch to establish social bonds with humans and facilitate cooperation in virtual reality (VR). We studied whether the emotional expression of a virtual agent would affect the way humans touch the agent. Participants were asked to hold a pressure-sensing tube presented as the agent's arm in VR. Upon seeing the agent's emotional expression change, participants briefly squeezed the arm. The effect of emotional expressions on affective state was measured using self-reported valence and arousal as well as physiology-based indices. Onset, duration, and intensity of the squeeze were recorded to examine the haptic responses. Emotional expression of agents affected squeeze intensity and duration through changes in emotional perception and experience. Haptic responses may thus provide an implicit measure of persons' experience towards their virtual companion.Peer reviewe

Evoking Physiological Synchrony and Empathy Using Social VR with Biofeedback

With the advent of consumer grade virtual reality (VR) headsets and physiological measurement devices, new possibilities for mediated social interaction emerge enabling the immersion to environments where the visual features react to the users' physiological activation. In this study, we investigated whether and how individual and interpersonally shared biofeedback (visualised respiration rate and frontal asymmetry of electroencephalography, EEG) enhance synchrony between the users' physiological activity and perceived empathy towards the other during a compassion meditation exercise carried out in a social VR setting. The study was conducted as a laboratory experiment (N = 72) employing a Unity3D-based Dynecom immersive social meditation environment and two amplifiers to collect the psychophysiological signals for the biofeedback. The biofeedback on empathy-related EEG frontal asymmetry evoked higher self-reported empathy towards the other user than the biofeedback on respiratory activation, but the perceived empathy was highest when both feedbacks were simultaneously presented. In addition, the participants reported more empathy when there was stronger EEG frontal asymmetry synchronization between the users. The presented results inform the field of affective computing on the possibilities that VR offers for different applications of empathic technologies.Peer reviewe

Digimuutoksessa onnistumisen eväät

Tämä dokumentti esittää Digimuutos-hankkeen tulokset, painottuen suosituksiin ja kokeilun esittelyyn. Hankkeen tavoitteena oli selvittää digitalisaation vaikutuksia työllisyyteen ja tarvittaviin osaamisiin. Dokumentin alussa esitetään lyhyesti aiemmin julkaistun kirjallisuuskatsauksen tulokset. Dokumentissa esitellään suosituksia neljään aihealueeseen: 1) Suomen vahvuudet ja tavoitetila, 2) Yritysten toiminnan edistäminen, 3) Tasa-arvosta huolehtiminen ja 4) Teknologian ja ihmisen yhteistyö. Laajasti esillä olleiden digitalisaation tuomien uhkakuvien toteutuminen voidaan torjua tehokkaimmin pyrkimällä edelläkävijäksi digitalisaation tehokkaassa ja innovatiivisessa hyödyntämisessä sekä tekemällä tarvittavia uudistuksia yhteiskunnan kaikilla tasoilla huomioiden käynnissä oleva suuri muutos. Suuri osa dokumenttia keskittyy Digimuutos-hankkeessa tehdyn kokeilun toteutuksen ja tulosten esittelyyn. Tehdyllä kokeilulla oli kolmen tasoisia tavoitteita: Ensinnäkin haluttiin saada selville se, auttaako työnhakuun liittyvien tiedollisten valmiuksien parantaminen työllistymistä. Toiseksi haluttiin selvittää kevyesti, onko digitalisaatioon liittyvillä valmiuksilla jokin yhteys työllistymiseen. Kolmanneksi tutkittiin sitä, millaisia haasteita ja ongelmia kevyestä valtionhallinnon parissa tehtävästä kokeiluasetelmasta syntyy ja kuinka ongelmat saadaan ratkaistua. Hankkeen kokeilu osoitti, että työnhaun tiedollisten valmiuksien lisääminen ei lisännyt työllistymistä. Toinen keskeinen anti Digimuutos-kokeilussa oli oppia siitä, millä tavalla ylipäätään hallinnon kokeiluja kannattaa tehdä. Digimuutos-kokeilu osoitti, että kokeiluasetelmaa toistamalla on mahdollista tehdä varsin kustannustehokkaasti testejä muualla käyttöön otettujen toimintamallien tai kokeilujen hyödyist

Increasing self–other similarity modulates ethnic bias in sensorimotor resonance to others’ pain

The tendency to simulate the pain of others within our own sensorimotor systems is a vital component of empathy. However, this sensorimotor resonance is modulated by a multitude of social factors including similarity in bodily appearance, e.g. skin colour. The current study investigated whether increasing self–other similarity via virtual transfer to another colour body reduced ingroup bias in sensorimotor resonance. A sample of 58 white participants was momentarily transferred to either a black or a white body using virtual reality technology. We then employed electroencephalography to examine event-related desynchronization (ERD) in the sensorimotor beta (13–23 Hz) oscillations while they viewed black, white and violet photorealistic virtual agents being touched with a noxious or soft object. While the noxious treatment of a violet agent did not increase beta ERD, amplified beta ERD in response to black agent’s noxious vs soft treatment was found in perceivers transferred to a black body. Transfer to the white body dismissed the effect. Further exploratory analysis implied that the pain-related beta ERD occurred only when the agent and the participant were of the same colour. The results suggest that even short-lasting changes in bodily resemblance can modulate sensorimotor resonance to others’ perceived pain.Peer reviewe