Virtual character facial expressions influence human brain and facial EMG activity in a decision-making game

We examined the effects of the emotional facial expressions of a virtual character (VC) on human frontal electroencephalographic (EEG) asymmetry (putatively indexing approach/withdrawal motivation), facial electromyographic (EMG) activity (emotional expressions), and social decision making (cooperation/defection). In a within-subjects design, the participants played the Iterated Prisoner's Dilemma game with VCs with different dynamic facial expressions (predefined or dependent on the participant's electrodermal and facial EMG activity). In general, VC facial expressions elicited congruent facial muscle activity. However, both frontal EEG asymmetry and facial EMG activity elicited by an angry VC facial expression varied as a function of preceding interactional events (human collaboration/defection). Pre-decision inner emotional-motivational processes and emotional facial expressions were dissociated, suggesting that human goals influence pre-decision frontal asymmetry, whereas display rules may affect (pre-decision) emotional expressions in human-VC interaction. An angry VC facial expression, high pre-decision corrugator EMG activity, and relatively greater left frontal activation predicted the participant's decision to defect. Both post-decision frontal asymmetry and facial EMG activity were related to reciprocal cooperation. The results suggest that the justifiability of VC emotional expressions and the perceived fairness of VC actions influence human emotional responses.Peer reviewe

Toward a second-person neuroscience

LS & BT : equal contributions (shared first-authorship)Peer reviewedPreprin

Affective games:a multimodal classification system

Affective gaming is a relatively new field of research that exploits human emotions to influence gameplay for an enhanced player experience. Changes in player’s psychology reflect on their behaviour and physiology, hence recognition of such variation is a core element in affective games. Complementary sources of affect offer more reliable recognition, especially in contexts where one modality is partial or unavailable. As a multimodal recognition system, affect-aware games are subject to the practical difficulties met by traditional trained classifiers. In addition, inherited game-related challenges in terms of data collection and performance arise while attempting to sustain an acceptable level of immersion. Most existing scenarios employ sensors that offer limited freedom of movement resulting in less realistic experiences. Recent advances now offer technology that allows players to communicate more freely and naturally with the game, and furthermore, control it without the use of input devices. However, the affective game industry is still in its infancy and definitely needs to catch up with the current life-like level of adaptation provided by graphics and animation

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

Touching virtual humans: Haptic responses reveal the emotional impact of affective agents

Interpersonal touch is critical for social-emotional development and presents a powerful modality for communicating emotions. Virtual agents of the future could capitalize on touch to establish social bonds with humans and facilitate cooperation in virtual reality (VR). We studied whether the emotional expression of a virtual agent would affect the way humans touch the agent. Participants were asked to hold a pressure-sensing tube presented as the agent's arm in VR. Upon seeing the agent's emotional expression change, participants briefly squeezed the arm. The effect of emotional expressions on affective state was measured using self-reported valence and arousal as well as physiology-based indices. Onset, duration, and intensity of the squeeze were recorded to examine the haptic responses. Emotional expression of agents affected squeeze intensity and duration through changes in emotional perception and experience. Haptic responses may thus provide an implicit measure of persons' experience towards their virtual companion.Peer reviewe

Tunteiden ja sosiaalisten tekijöiden vaikutus pelikokemukseen

Arguably, the emotions elicited by playing are the reason why people play digital games. Social interaction is an important source of emotion during game play, but research on it is rather sparse. In this dissertation I briefly review the emotion-theoretic literature in order to better understand what emotion means in the context of games, and how this should be taken into account when measuring emotions related to a game experience. Study I presents a review of the use of psychophysiological methods in game research. I show that the theoretical background behind these methods generally tends to be neglected. This could be remedied by a theoretical framework that integrates the understanding of emotions and explicitly describes the links between different emotion measures and the theoretical concepts they are professed to reflect. I present my proposition for the first step towards such a framework in Study II. I employ the sociality characteristics framework by de Kort and IJsselsteijn (2008) and my interpretation of the social factors in order to study the effect of the central social context factors on the emotional game experience. Study III presents evidence that in addition to tonic physiological levels, the relationship between the participants also affects the momentary, phasic responses to the key game events victory and defeat. In particular, although physiological signals can, to a certain extent, be used to assess emotional experiences (such as positive responses to a victory), in some cases the typical psychophysiological mappings may even be completely opposite. Interpreting these signals requires a broader theoretical understanding than what is typically acknowledged. Study IV supports the earlier findings that competition is experienced more positively than cooperation but that the effect is dependent on gender, as this was found only in males. For females, there was no difference between the two modes, and no difference in negative activation. In addition, self-reports concerning social presence suggested that this concept is not always associated with higher positive emotions, while a form of friendly rivalry (associated with lower social presence) might be experienced positively a finding apparently new in existing literature. Finally, Study V provides insight into the practical significance of the measurements with a predictive validity study, showing practical effects how the certain kinds of game experiences may lead to greater game use and preference, but that these links are not as simple as previously suggested. In sum, this work offers new knowledge on how social context factors are generally related to the game experience, on how emotions can be studied in game research and what theoretical considerations should be taken into account, and on the emotional effects of particular social context factors during play. The results are mainly useful for further basic game research, but they have also potential implications for general emotion research, the game industry, and in the long run, society at large.Digitaalisten pelien herättämiä tunnekokemuksia on pidetty tärkeänä selittäjänä näiden suosiolle. Sosiaalinen vuorovaikutus pelin aikana taas on merkittävä tunnekokemuksien lähde, mutta sitä on tutkittu melko vähän. Tässä väitöskirjassa käyn osaksi läpi tunteita selittävää emootioteoriakirjallisuutta tarkastellakseni mitä tunteen tai emootion käsite tarkoittaa pelikontekstissa ja kuinka tämä tieto tulisi ottaa huomioon kun tunnekokemusta halutaan mitata pelitutkimuksessa. Tutkimus I esittää katsauksen psykofysiologisten menetelmien käytöstä pelitutkimuksessa, jonka avulla osoitan kuinka näiden menetelmien taustalla oleva teoreettinen perusta jätetään usein huomiotta. Tähän tyypilliseen puutteeseen olisi avuksi kokonaisvaltainen teoreettinen emootioiden ja niiden mittaamisen viitekehys. Esitän oman ehdotukseni kyseisenlaisen viitekehyksen suuntaan Tutkimuksessa II. Käytän tutkimukseni empiirisessä osuudessa tulkintaani de Kortin ja IJsselsteijnin (2008) teoreettisesta viitekehyksestä sosiaalisuuspiirteistä tutkiakseni merkittävien sosiaalisten tekijöiden merkitystä pelaamisen tunnekokemukselle. Tutkimuksessa III esitän todisteita, että pelaajien väliset suhteet vaikuttavat pitkien jaksojen keskiarvojen lisäksi myös yksittäisten tilanteiden voiton ja häviön laukaisemiin hetkellisiin reaktioihin. Tulokset osoittavat kuitenkin, että joissain tilanteissa reaktiot voivat olla odottamattomat, minkä vuoksi fysiologisten mittausten tulkinnassa tulisi käyttää laajempaa teoreettista ymmärrystä kuin on yleistä. Tutkimus IV:n tulokset tukevat aiempia löydöksiä, että peleissä kilpailu koetaan positiivisemmin kuin yhteistyö, mutta että tämä vaikutus riippuu sukupuolesta: tulos pätee vain miehiin, kun taas naiset eivät osoittaneet eroa näiden pelimuotojen välillä positiivisen eivätkä negatiivisen tunnereaktion suhteen. Lisäksi huomattiin, että sosiaalisen läsnäolon kokemuksella, joka on yleensä yhdistetty positiiviseen kokemukseen, ei ole selvää yhteyttä positiivisuuteen, kun taas emootiokirjallisuudessa esitetyn selityksen vastaisesti tietynlainen ystävällismielinen vahingonilo voidaan kokea positiivisena. Lopuksi, Tutkimus V esittää kuinka emootiomittauksilla voi olla käytännöllistä ennustevaliditeettia, missä tietynlaiset pelikokemukset voidaan yhdistää tulevaan pelikäyttäytymiseen, mutta että nämä yhteydet eivät ole niin yksinkertaisia kuin on aiemmin esitetty. Yhteenvetona, työni tarjoaa uutta tietoa siitä, mikä on sosiaalisten taustatekijöiden yleinen yhteys pelikokemukseen, kuinka emootioita voidaan tarkastella pelitutkimuksessa ja mitä teoreettisia näkökohtia tällöin tulisi ottaa huomioon, sekä siitä mitä ovat tiettyjen sosiaalisten taustatekijöiden emootiovaikutus pelikokemuksen aikana. Työn tuloksia voidaan käyttää lähinnä pelaamisen perustutkimuksessa, mutta niillä on myös mahdollista merkitystä yleisessä emootiotutkimuksessa, pelitutkimuksen käytännöllisessä soveltamisessa peliteollisuuden hyödyksi, sekä pitkällä aikavälillä myös laajempien yhteiskunnallisten kysymysten kannalta

Modulation of facial mimicry by attitudes

The current experiment explored the influence of attitudes on facial reactions to emotional faces. The participants’ attitudes (positive, neutral, and negative) towards three types of characters were manipulated by written reports. Afterwards participants saw happy, neutral, and sad facial expressions of the respective characters while their facial muscular reactions (M. Corrugator supercilii and M. Zygomaticus major) were recorded electromyografically. Results revealed facial mimicry reactions to happy and sad faces of positive characters, but less and even incongruent facial muscular reactions to happy and sad faces of negative characters. Overall, the results show that attitudes, formed in a few minutes, and only by reports and not by own experiences, can moderate automatic non-verbal social behavior, i.e. facial mimicry