Knowledge-based Framework for Intelligent Emotion Recognition in Spontaneous Speech

AbstractAutomatic speech emotion recognition plays an important role in intelligent human computer interaction. Identifying emotion in natural, day to day, spontaneous conversational speech is difficult because most often the emotion expressed by the speaker are not necessarily as prominent as in acted speech. In this paper, we propose a novel spontaneous speech emotion recognition framework that makes use of the available knowledge. The framework is motivated by the observation that there is significant disagreement amongst human annotators when they annotate spontaneous speech; the disagreement largely reduces when they are provided with additional knowledge related to the conversation. The proposed framework makes use of the contexts (derived from linguistic contents) and the knowledge regarding the time lapse of the spoken utterances in the context of an audio call to reliably recognize the current emotion of the speaker in spontaneous audio conversations. Our experimental results demonstrate that there is a significant improvement in the performance of spontaneous speech emotion recognition using the proposed framework

Long Term Suboxone™ Emotional Reactivity As Measured by Automatic Detection in Speech

Addictions to illicit drugs are among the nation’s most critical public health and societal problems. The current opioid prescription epidemic and the need for buprenorphine/naloxone (Suboxone®; SUBX) as an opioid maintenance substance, and its growing street diversion provided impetus to determine affective states (“true ground emotionality”) in long-term SUBX patients. Toward the goal of effective monitoring, we utilized emotion-detection in speech as a measure of “true” emotionality in 36 SUBX patients compared to 44 individuals from the general population (GP) and 33 members of Alcoholics Anonymous (AA). Other less objective studies have investigated emotional reactivity of heroin, methadone and opioid abstinent patients. These studies indicate that current opioid users have abnormal emotional experience, characterized by heightened response to unpleasant stimuli and blunted response to pleasant stimuli. However, this is the first study to our knowledge to evaluate “true ground” emotionality in long-term buprenorphine/naloxone combination (Suboxone™). We found in long-term SUBX patients a significantly flat affect (p<0.01), and they had less self-awareness of being happy, sad, and anxious compared to both the GP and AA groups. We caution definitive interpretation of these seemingly important results until we compare the emotional reactivity of an opioid abstinent control using automatic detection in speech. These findings encourage continued research strategies in SUBX patients to target the specific brain regions responsible for relapse prevention of opioid addiction.United States. Dept. of Defense. Assistant Secretary of Defense for Research & Engineering (Air Force Contract FA8721-05-C-0002

Advancing Fine-Grained Emotion Recognition in Short Text

Advanced emotion recognition in text is essential for developing intelligent affective applications, which can recognize, react upon, and analyze users' emotions. Our particular motivation for solving this problem lies in large-scale analysis of social media data, such as those generated by Twitter users. Summarizing users' emotions can enable better understandings of their reactions, interests, and motivations. We thus narrow the problem to emotion recognition in short text, particularly tweets. Another driving factor of our work is to enable discovering emotional experiences at a detailed, fine-grained level. While many researchers focus on recognizing a small number of basic emotion categories, humans experience a larger variety of distinct emotions. We aim to recognize as many as 20 emotion categories from the Geneva Emotion Wheel. Our goal is to study how to build such fine-grained emotion recognition systems. We start by surveying prior approaches to building emotion classifiers. The main body of this thesis studies two of them in detail: crowdsourcing and distant supervision. Based on them, we design fine-grained domain-specific systems to recognize users' reactions to sporting events captured on Twitter and address multiple challenges that arise in that process. Crowdsourcing allows extracting affective commonsense knowledge by asking hundreds of workers for manual annotation. The challenge is in collecting informative and truthful annotations. To address it, we design a human computation task that elicits both emotion category labels and emotion indicators (i.e. words or phrases indicative of labeled emotions). We also develop a methodology to build an emotion lexicon using such data. Our experiments show that the proposed crowdsourcing method can successfully generate a domain-specific emotion lexicon. Additionally, we suggest how to teach and motivate non-expert annotators. We show that including a tutorial and using carefully formulated reward descriptions can effectively improve annotation quality. Distant supervision consists of building emotion classifiers from data that are automatically labeled using some heuristics. This thesis studies heuristics that apply emotion lexicons of limited quality, for example due to missing or erroneous term-emotion associations. We show the viability of such an approach to obtain domain-specific classifiers having substantially better quality of recognition than the initial lexicon-based ones. Our experiments reveal that treating the emotion imbalance in training data and incorporating pseudo-neutral documents is crucial for such improvement. This method can be applied to building emotion classifiers across different domains using limited input resources and thus requiring minimal effort. Another challenge for lexicon-based emotion recognition is to reduce the error introduced by linguistic modifiers such as negation and modality. We design a data analysis method that allows modeling the specific effects of the studied modifiers, both in terms of shifting emotion categories and changing confidence in emotion presence. We show that the effects of modifiers vary across the emotion categories, which indicates the importance of treating such effects at a more fine-grained level to improve classification quality. Finally, the thesis concludes with our recommendations on how to address the examined general challenges of building a fine-grained textual emotion recognition system

Proceedings of the LREC 2018 Special Speech Sessions

LREC 2018 Special Speech Sessions "Speech Resources Collection in Real-World Situations"; Phoenix Seagaia Conference Center, Miyazaki; 2018-05-0

When in doubt ask the crowd : leveraging collective intelligence for improving event detection and machine learning

[no abstract

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

An evidence-based toolset to capture, measure, analyze & assess emotional health

This thesis describes the development and validation of an evidence-based toolkit that captures a patient’s emotional state, expressiveness/affect, self-awareness, and empathy during a fifteen second telephone call, and then accurately measures and analyzes these indicators of Emotional Health based on emotion detection in speech and multilevel regression analysis. An emotion corpus of eight thousand three hundred and seventy-six (8,376) momentary emotional states was collected from one hundred and thirteen (113) participants including three groups: Opioid Addicts undergoing Suboxone® treatment, the General Population, and members of Alcohol Anonymous. Each collected emotional state includes an emotional recording in response to “How are you feeling?” a self-assessment of emotional state, and an assessment of an emotionally-charged recording. Each recording is labeled with the emotional truth. A method for unsupervised emotional truth corpus labeling through automatic audio chunking and unsupervised automatic emotional truth labeling is proposed and experimented. In order to monitor and analyze the emotional health of a patient, algorithms are developed to accurately measure the emotional state of a patient in their natural environment. Real-time emotion detection in speech provides instantaneous classification of the emotional truth of a speech recording. A pseudo real-time method improves emotional truth accuracy as more data becomes available. A new measure of emotional truth accuracy, the certainty score, is introduced. Measures of self-awareness, empathy, and expressiveness are derived from the collected emotional state. Are there differences in emotional truth, self-assessment, self-awareness, and empathy across groups? Does gender have an effect? Does language have an effect? Does length of the response, as an indication of emotional expressiveness, vary with emotion or group? Does confidence of the emotional label, as an indication of affect, vary with emotion or group? Are there differences in call completion rates? Which group would be more likely to continue in data collections? Significant results to these questions will provide evidence that capturing and measuring Emotional Health in speech can: Assist therapists and patients in Cognitive Behavioural Therapy to become aware of symptoms and make it easier to change thoughts and behaviours; Provide evidence of psychotropic medication and psychotherapy effectiveness in mental health and substance abuse treatment programs; Accelerate the interview process during monthly assessments by physicians, psychiatrists, and therapists by providing empirical insight into emotional health of patients in their natural environment. Trigger crisis intervention on conditions including the detection of isolation from unanswered calls, or consecutive days of negative emotions