Towards Computer-Assisted Regulation of Emotions

Tunteet ovat keskeinen ja erottamaton osa ihmisen toimintaa, ajattelua ja yksilöiden välistä vuorovaikutusta. Tunteet luovat perustan mielekkäälle, toimivalle ja tehokkaalle toiminnalle. Joskus tunteiden sävy tai voimakkuus voi kuitenkin olla epäedullinen henkilön tavoitteiden ja hyvinvoinnin kannalta. Tällöin taidokas tunteiden säätely voi auttaa saavuttamaan terveen ja menestyksellisen elämän. Väitöstyön tavoitteena oli muodostaa perusta tulevaisuuden tietokoneille, jotka auttavat säätelemään tunteita. Tietokoneiden tunneälyä on toistaiseksi kehitetty kahdella alueella: ihmisen tunnereaktioiden mittaamisessa ja tietokoneen tuottamissa tunneilmaisuissa. Viimeisimmät teknologiat antavat tietokoneille jo mahdollisuuden tunnistaa ja jäljitellä ihmisen tunneilmaisuja hyvinkin tarkasti. Väitöstyössä toimistotuoliin asennetuilla paineantureilla kyettiin huomaamattomasti havaitsemaan muutoksia kehon liikkeissä: osallistujat nojautuivat kohti heille esitettyjä tietokonehahmoja. Tietokonehahmojen esittämät kasvonilmeet ja kehollinen etäisyys vaikuttivat merkittävästi osallistujien tunne- ja tarkkaavaisuuskokemuksiin sekä sydämen, ihon hikirauhasten ja kasvon lihasten toimintaan. Tulokset osoittavat että keinotekoiset tunneilmaisut voivat olla tehokkaita henkilön kokemusten ja kehon toiminnan säätelyssä. Väitöstyössä laadittiin lopulta vuorovaikutteinen asetelma, jossa tunneilmaisujen automaattinen tarkkailu liitettiin tietokoneen tuottamien sosiaalisten ilmaisujen ohjaamiseen. Osallistujat pystyivät säätelemään välittömiä fysiologisia reaktioitaan ja tunnekokemuksiaan esittämällä tahdonalaisia kasvonilmeitä (mm. ikään kuin hymyilemällä) heitä lähestyvälle tietokonehahmolle. Väitöstyön tuloksia voidaan hyödyntää laajasti, muun muassa uudenlaisten, ihmisen luonnollisia vuorovaikutustapoja paremmin tukevien tietokoneiden suunnittelussa.Emotions are intimately connected with our lives. They are essential in motivating behaviour, for reasoning effectively, and in facilitating interactions with other people. Consequently, the ability to regulate the tone and intensity of emotions is important for leading a life of success and well-being. Intelligent computer perception of human emotions and effective expression of virtual emotions provide a basis for assisting emotion regulation with technology. State-of-the-art technologies already allow computers to recognize and imitate human social and emotional cues accurately and in great detail. For example, in the present work a regular looking office chair was used to covertly measure human body movement responses to artifical expressions of proximity and facial cues. In general, such artificial cues from visual agents were found to significantly affect heart, sweat gland, and facial muscle activities, as well as subjective experiences of emotion and attention. The perceptual and expressive capabilities were combined in a setup where a person regulated her or his more spontaneous reactions by either smiling or frowning voluntarily to a virtual humanlike character. These results highlight the potential of future emotion-sensitive technologies for creating supportive and even healthy interactions between humans and computers

Vocal emotions on the brain: the role of acoustic parameters and musicality

The human voice is a powerful transmitter of emotions. This dissertation addresses three main gaps in the field of vocal emotion perception. The first is the quantification of the relative contribution of fundamental frequency (F0) and timbre cues to the perception of different emotions and their associated electrophysiological correlates. Using parameter-specific voice morphing, the results show that both F0 and timbre carry unique information that allow emotional inferences, although F0 seems to be relatively more important overall. The electrophysiological data revealed F0- and timbre-specific modulations in several ERP components, such as the P200 and the N400. Second, it was explored how musicality affects the processing of emotional voice cues, by providing a review on the literature linking musicality to emotion perception and subsequently showing that musicians have a benefit in vocal emotion perception compared to non-musicians. The present data offer original insight into the special role of pitch cues: musicians outperformed non-musicians when emotions were expressed by the pitch contour only, but not when they were expressed by vocal timbre. Although the electrophysiological patterns were less conclusive, they imply that musicality may modulate brain responses to vocal emotions. Third, this work provides a critical reflection on parameter-specific voice morphing and its suitability to study the processing of vocal emotions. Distortions in voice naturalness resulting from extreme acoustic manipulations were identified as one of the major threats to the ecological validity of the stimulus material produced with this technique. However, the results suggested that while voice morphing does affect the perceived naturalness of stimuli, behavioral measures of emotion perception were found to be remarkably robust against these distortions. Thus, the present data advocate parameter-specific voice morphing as a valid tool for vocal emotional research

The Perception of Emotion from Acoustic Cues in Natural Speech

Knowledge of human perception of emotional speech is imperative for the development of emotion in speech recognition systems and emotional speech synthesis. Owing to the fact that there is a growing trend towards research on spontaneous, real-life data, the aim of the present thesis is to examine human perception of emotion in naturalistic speech. Although there are many available emotional speech corpora, most contain simulated expressions. Therefore, there remains a compelling need to obtain naturalistic speech corpora that are appropriate and freely available for research. In that regard, our initial aim was to acquire suitable naturalistic material and examine its emotional content based on listener perceptions. A web-based listening tool was developed to accumulate ratings based on large-scale listening groups. The emotional content present in the speech material was demonstrated by performing perception tests on conveyed levels of Activation and Evaluation. As a result, labels were determined that signified the emotional content, and thus contribute to the construction of a naturalistic emotional speech corpus. In line with the literature, the ratings obtained from the perception tests suggested that Evaluation (or hedonic valence) is not identified as reliably as Activation is. Emotional valence can be conveyed through both semantic and prosodic information, for which the meaning of one may serve to facilitate, modify, or conflict with the meaning of the other—particularly with naturalistic speech. The subsequent experiments aimed to investigate this concept by comparing ratings from perception tests of non-verbal speech with verbal speech. The method used to render non-verbal speech was low-pass filtering, and for this, suitable filtering conditions were determined by carrying out preliminary perception tests. The results suggested that nonverbal naturalistic speech provides sufficiently discernible levels of Activation and Evaluation. It appears that the perception of Activation and Evaluation is affected by low-pass filtering, but that the effect is relatively small. Moreover, the results suggest that there is a similar trend in agreement levels between verbal and non-verbal speech. To date it still remains difficult to determine unique acoustical patterns for hedonic valence of emotion, which may be due to inadequate labels or the incorrect selection of acoustic parameters. This study has implications for the labelling of emotional speech data and the determination of salient acoustic correlates of emotion

Evaluating embodied conversational agents in multimodal interfaces

Based on cross-disciplinary approaches to Embodied Conversational Agents, evaluation methods for such human-computer interfaces are structured and presented. An introductory systematisation of evaluation topics from a conversational perspective is followed by an explanation of social-psychological phenomena studied in interaction with Embodied Conversational Agents, and how these can be used for evaluation purposes. Major evaluation concepts and appropriate assessment instruments – established and new ones – are presented, including questionnaires, annotations and log-files. An exemplary evaluation and guidelines provide hands-on information on planning and preparing such endeavours

Attitudes of Communication Partners Toward AAC Users

The purpose of the current study was to investigate the relationship between the attitudes of communication partners of augmentative and alternative communication (AAC) users and device voice output (specifically the Tango! and DV4). Also, the study examined the relationship between attitudes and age of the communication partner of the AAC user. Six children, twenty-five pre-service Speech-Language Pathologists, and four in-service Speech- Language Pathologists familiar with AAC watched videos of an interaction between a child using AAC to communication with an adult. Following the videos, each participant responded to an attitude scale questionnaire. No significant effect was found for the child participant attitudes toward the Tango! and DV4. However, significant results were found between the devices for pre-service adults, in-service adults, and adults as a whole. Recommendations and clinical implications are suggested

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Emotional reactions to music: psychophysiological correlates and applications to affective disorders

Music has been used to evoke emotions for centuries. The mechanisms underlying this effect have remained largely unclear. This thesis contributes to research on how music evokes emotions by investigating two mechanisms from the model of Juslin and Västfjäll (2008) - musical expectancy and emotional contagion. In the perception studies the focus is on how musical expectancy violations are detected by either musically trained or untrained individuals. In the music-making studies, we concentrate on mood change brought about by cheerful music in healthy and depressed individuals and factors which could modulate this change like personality, musical preference and general emotional state. The results indicate that the subtlest scale violations are detected at the level of brain electrical potential while the task remains behaviourally difficult. This suggests that scale information is processed using music-syntactic analysis and in memory existing representations of tonal hierarchies, instead of auditory sensory memory as previously believed. Music-making decreased anxiety, depression and fatigue in both depressed and healthy participants whereas arousal and positive mood increased. This suggests that musicmaking could be beneficial for depressed individuals in terms of improving their mood on a short-term basis, even though a reliable music-related decrease of depression symptoms was not found. Among healthy participants, intraversion, openness to experience, agreeableness, musical training and liking of the music predicted positive changes in mood following a music-making session. Taken together, these studies indicate that subtle musical scale violations are detected even if they are not consciously perceived as deviants and could therefore be used to evoke emotions, and music-making improves the mood in both healthy and depressed individuals and could serve as a temporary relief in case of depression

The effects of user assistance systems on user perception and behavior

The rapid development of information technology (IT) is changing how people approach and interact with IT systems (Maedche et al. 2016). IT systems can increasingly support people in performing ever more complex tasks (Vtyurina and Fourney 2018). However, people's cognitive abilities have not evolved as quickly as technology (Maedche et al. 2016). Thus, different external factors (e.g., complexity or uncertainty) and internal conditions (e.g., cognitive load or stress) reduce decision quality (Acciarini et al. 2021; Caputo 2013; Hilbert 2012). User-assistance systems (UASs) can help to compensate for human weaknesses and cope with new challenges. UASs aim to improve the user's cognition and capabilities, benefiting individuals, organizations, and society. To achieve this goal, UASs collect, prepare, aggregate, analyze information, and communicate results according to user preferences (Maedche et al. 2019). This support can relieve users and improve the quality of decision-making. Using UASs offers many benefits but requires successful interaction between the user and the UAS. However, this interaction introduces social and technical challenges, such as loss of control or reduced explainability, which can affect user trust and willingness to use the UAS (Maedche et al. 2019). To realize the benefits, UASs must be developed based on an understanding and incorporation of users' needs. Users and UASs are part of a socio-technical system to complete a specific task (Maedche et al. 2019). To create a benefit from the interaction, it is necessary to understand the interaction within the socio-technical system, i.e., the interaction between the user, UAS, and task, and to align the different components. For this reason, this dissertation aims to extend the existing knowledge on UAS design by better understanding the effects and mechanisms during the interaction between UASs and users in different application contexts. Therefore, theory and findings from different disciplines are combined and new theoretical knowledge is derived. In addition, data is collected and analyzed to validate the new theoretical knowledge empirically. The findings can be used to reduce adaptation barriers and realize a positive outcome. Overall this dissertation addresses the four classes of UASs presented by Maedche et al. (2016): basic UASs, interactive UASs, intelligent UASs, and anticipating UASs. First, this dissertation contributes to understanding how users interact with basic UASs. Basic UASs do not process contextual information and interact little with the user (Maedche et al. 2016). This behavior makes basic UASs suitable for application contexts, such as social media, where little interaction is desired. Social media is primarily used for entertainment and focuses on content consumption (Moravec et al. 2018). As a result, social media has become an essential source of news but also a target for fake news, with negative consequences for individuals and society (Clarke et al. 2021; Laato et al. 2020). Thus, this thesis presents two approaches to how basic UASs can be used to reduce the negative influence of fake news. Firstly, basic UASs can provide interventions by warning users of questionable content and providing verified information but the order in which the intervention elements are displayed influences the fake news perception. The intervention elements should be displayed after the fake news story to achieve an efficient intervention. Secondly, basic UASs can provide social norms to motivate users to report fake news and thereby stop the spread of fake news. However, social norms should be used carefully, as they can backfire and reduce the willingness to report fake news. Second, this dissertation contributes to understanding how users interact with interactive UASs. Interactive UASs incorporate limited information from the application context but focus on close interaction with the user to achieve a specific goal or behavior (Maedche et al. 2016). Typical goals include more physical activity, a healthier diet, and less tobacco and alcohol consumption to prevent disease and premature death (World Health Organization 2020). To increase goal achievement, previous researchers often utilize digital human representations (DHRs) such as avatars and embodied agents to form a socio-technical relationship between the user and the interactive UAS (Kim and Sundar 2012a; Pfeuffer et al. 2019). However, understanding how the design features of an interactive UAS affect the interaction with the user is crucial, as each design feature has a distinct impact on the user's perception. Based on existing knowledge, this thesis highlights the most widely used design features and analyzes their effects on behavior. The findings reveal important implications for future interactive UAS design. Third, this dissertation contributes to understanding how users interact with intelligent UASs. Intelligent UASs prioritize processing user and contextual information to adapt to the user's needs rather than focusing on an intensive interaction with the user (Maedche et al. 2016). Thus, intelligent UASs with emotional intelligence can provide people with task-oriented and emotional support, making them ideal for situations where interpersonal relationships are neglected, such as crowd working. Crowd workers frequently work independently without any significant interactions with other people (Jäger et al. 2019). In crowd work environments, traditional leader-employee relationships are usually not established, which can have a negative impact on employee motivation and performance (Cavazotte et al. 2012). Thus, this thesis examines the impact of an intelligent UAS with leadership and emotional capabilities on employee performance and enjoyment. The leadership capabilities of the intelligent UAS lead to an increase in enjoyment but a decrease in performance. The emotional capabilities of the intelligent UAS reduce the stimulating effect of leadership characteristics. Fourth, this dissertation contributes to understanding how users interact with anticipating UASs. Anticipating UASs are intelligent and interactive, providing users with task-related and emotional stimuli (Maedche et al. 2016). They also have advanced communication interfaces and can adapt to current situations and predict future events (Knote et al. 2018). Because of these advanced capabilities anticipating UASs enable collaborative work settings and often use anthropomorphic design cues to make the interaction more intuitive and comfortable (André et al. 2019). However, these anthropomorphic design cues can also raise expectations too high, leading to disappointment and rejection if they are not met (Bartneck et al. 2009; Mori 1970). To create a successful collaborative relationship between anticipating UASs and users, it is important to understand the impact of anthropomorphic design cues on the interaction and decision-making processes. This dissertation presents a theoretical model that explains the interaction between anthropomorphic anticipating UASs and users and an experimental procedure for empirical evaluation. The experiment design lays the groundwork for empirically testing the theoretical model in future research. To sum up, this dissertation contributes to information systems knowledge by improving understanding of the interaction between UASs and users in different application contexts. It develops new theoretical knowledge based on previous research and empirically evaluates user behavior to explain and predict it. In addition, this dissertation generates new knowledge by prototypically developing UASs and provides new insights for different classes of UASs. These insights can be used by researchers and practitioners to design more user-centric UASs and realize their potential benefits

Sampling inner experience in the learning disabled population

The inner experience of four learning disabled individuals and three undiagnosed individuals, ages 24 to 40 years, was investigated using the method of descriptive inner experience sampling (Hurlburt, 1990). This method requires that the subject freeze moments of inner experience when signaled by a randomly programmed beeping device, and write down the thoughts, feelings, images and/or other inner experiences occurring at that precise moment. Within 24 hours, the subject met with the experimenter and discussed in detail each sampled moment. The experimenter then prepared descriptions of each sampled moment, which were examined for salient characteristics which emerged from these descriptions. It was found that learning disabled individuals experience considerably less Inner Speech than most normal individuals, supporting the findings of other research that did not employ the thought sampling method. The learning disabled subjects also experienced Images, Feelings, and Unsymbolized Thinking more frequently than normal subjects. Three undiagnosed subjects were also examined in this manner. These subjects do not form a control group but are presented to add to the pool of subjects that have been sampled using this method