Emotion Recognition from Acted and Spontaneous Speech

Dizertační práce se zabývá rozpoznáním emočního stavu mluvčích z řečového signálu. Práce je rozdělena do dvou hlavních častí, první část popisuju navržené metody pro rozpoznání emočního stavu z hraných databází. V rámci této části jsou představeny výsledky rozpoznání použitím dvou různých databází s různými jazyky. Hlavními přínosy této části je detailní analýza rozsáhlé škály různých příznaků získaných z řečového signálu, návrh nových klasifikačních architektur jako je například „emoční párování“ a návrh nové metody pro mapování diskrétních emočních stavů do dvou dimenzionálního prostoru. Druhá část se zabývá rozpoznáním emočních stavů z databáze spontánní řeči, která byla získána ze záznamů hovorů z reálných call center. Poznatky z analýzy a návrhu metod rozpoznání z hrané řeči byly využity pro návrh nového systému pro rozpoznání sedmi spontánních emočních stavů. Jádrem navrženého přístupu je komplexní klasifikační architektura založena na fúzi různých systémů. Práce se dále zabývá vlivem emočního stavu mluvčího na úspěšnosti rozpoznání pohlaví a návrhem systému pro automatickou detekci úspěšných hovorů v call centrech na základě analýzy parametrů dialogu mezi účastníky telefonních hovorů.Doctoral thesis deals with emotion recognition from speech signals. The thesis is divided into two main parts; the first part describes proposed approaches for emotion recognition using two different multilingual databases of acted emotional speech. The main contributions of this part are detailed analysis of a big set of acoustic features, new classification schemes for vocal emotion recognition such as “emotion coupling” and new method for mapping discrete emotions into two-dimensional space. The second part of this thesis is devoted to emotion recognition using multilingual databases of spontaneous emotional speech, which is based on telephone records obtained from real call centers. The knowledge gained from experiments with emotion recognition from acted speech was exploited to design a new approach for classifying seven emotional states. The core of the proposed approach is a complex classification architecture based on the fusion of different systems. The thesis also examines the influence of speaker’s emotional state on gender recognition performance and proposes system for automatic identification of successful phone calls in call center by means of dialogue features.

Multiscale Contextual Learning for Speech Emotion Recognition in Emergency Call Center Conversations

Emotion recognition in conversations is essential for ensuring advanced human-machine interactions. However, creating robust and accurate emotion recognition systems in real life is challenging, mainly due to the scarcity of emotion datasets collected in the wild and the inability to take into account the dialogue context. The CEMO dataset, composed of conversations between agents and patients during emergency calls to a French call center, fills this gap. The nature of these interactions highlights the role of the emotional flow of the conversation in predicting patient emotions, as context can often make a difference in understanding actual feelings. This paper presents a multi-scale conversational context learning approach for speech emotion recognition, which takes advantage of this hypothesis. We investigated this approach on both speech transcriptions and acoustic segments. Experimentally, our method uses the previous or next information of the targeted segment. In the text domain, we tested the context window using a wide range of tokens (from 10 to 100) and at the speech turns level, considering inputs from both the same and opposing speakers. According to our tests, the context derived from previous tokens has a more significant influence on accurate prediction than the following tokens. Furthermore, taking the last speech turn of the same speaker in the conversation seems useful. In the acoustic domain, we conducted an in-depth analysis of the impact of the surrounding emotions on the prediction. While multi-scale conversational context learning using Transformers can enhance performance in the textual modality for emergency call recordings, incorporating acoustic context is more challenging

The Role of Customer Emotions in a Call Center Setting

This thesis investigates the role of customer emotions in a call center setting. Three empirical studies are presented: 1. An exploratory qualitative study shows that emotions play an important role in a call center encounter. A measurement scale is developed using the negative emotions reported: frustration, anger, powerlessness, discouragement and stress, and positive emotions: happiness and relief. 2. A survey of 1440 customers of a French call center and subsequent Structural Equation Modeling shows that emotions have a significant effect on satisfaction and loyalty. 3. A follow-up qualitative study indicates that emotions change during the process of a call, and are accompanied by appraisals of reduced certainty and control. Frustration is shown to be a major customer emotional experience due to the set-up of call center services which generate reduced certainty and control, attributions of employee or company responsibility and appraisals of goal-blocking. The main theoretical contributions come from a) the finding that emotions have a stronger effect on loyalty than does satisfaction, b) empirical testing of a widely acknowledged (but not explicitly tested) notion that negative emotions decrease loyalty more than positive emotions increase it. In a call center context callers begin the call with some negative emotional ‘baggage’ which may enhance the effect on loyalty as compared to positive emotions, c) empirical support for Barrett et al. (2007) and Elfenbein (2007) who suggest that an emotional process characterizes emotional experience better than emotions alone. Managerial implications are a) that the call center industry needs to move away from its traditional performance-metrics basis for measuring customer satisfaction and take customer emotions into account, and b) that to ensure loyalty call center managers should focus more strongly on strategies for avoiding negative customer emotional states than for promoting positive emotions

A review of natural language processing in contact centre automation

Contact centres have been highly valued by organizations for a long time. However, the COVID-19 pandemic has highlighted their critical importance in ensuring business continuity, economic activity, and quality customer support. The pandemic has led to an increase in customer inquiries related to payment extensions, cancellations, and stock inquiries, each with varying degrees of urgency. To address this challenge, organizations have taken the opportunity to re-evaluate the function of contact centres and explore innovative solutions. Next-generation platforms that incorporate machine learning techniques and natural language processing, such as self-service voice portals and chatbots, are being implemented to enhance customer service. These platforms offer robust features that equip customer agents with the necessary tools to provide exceptional customer support. Through an extensive review of existing literature, this paper aims to uncover research gaps and explore the advantages of transitioning to a contact centre that utilizes natural language solutions as the norm. Additionally, we will examine the major challenges faced by contact centre organizations and offer reco

1-800-(Re)Colonize: A Feminist Postcolonial and Performance Analysis of Call Center Agents in India Performing U.S. Cultural Identity

The contemporary historical moment finds us in a web of globalization that spans the globe. While our interconnectedness brings us into unforeseen communications, we enter the conversation grounded in particular subject locations. Postcolonial subjectivities hold strategic memories of colonial violences as a means of survival and resistance while colonizing forces hold onto binary narratives of their own superiority. Globalization provides the context wherein decolonized and colonizing nations interact with unequal power resulting in multifaceted outcomes, one of which I argue is a re-colonial dynamic. The phenomenon of U.S. corporate outsourcing to India is one instance where a re-colonial dynamic occurs. India\u27s post-1991 liberalization policies facilitated its current relationship with U.S. corporations, many of which invested heavily in India\u27s economy and telecommunications development. One facet of this investment resulted in the creation of call centers which provide customer service support to large corporations. Indian call centers supply customer service operations to U.S. corporations and Indian workers interact with U.S. consumers on the telephone. The condition of employment for largely 20- to 30-something Indian workers, what marks the unequal power relations and re-colonial dynamic, is a performance of American culture. Indian call center agents undergo training in American voice and culture to mimic and interact with the U.S. consumer while simultaneously erasing their Indian cultural identities. To understand the implications of this practice, I rely on the voices of Indian call center agents and their performance of U.S. culture in their work and training and its impact on their daily and cultural lives. The performances come from personal interviews with call center agents conducted by Sheena Malhotra and me in Bangalore and Mumbai, India, on film footage from Aradhana Seth\u27s documentary I-800-CALLRVDIA, and on media representations from U.S. mainstream media. Interweaving postcolonial and performance theories as the framework, I use Robert Scholes (1985) method of textual criticism which involves a three-step hermeneutic process of reading, interpreting and criticizing performances to deconstruct and analyze their pleasures and power. I rely on Homi K. Bhabha\u27s (1 994) theorization of ambivalence, hybridity and mimicry to understand colonial subjects\u27 complex negotiation of colonial forces. From these performances emerge several themes and reveal the tensions between colonial forces of corporations and the complex negotiations of it through the performances of postcolonial subjectivities. While U.S. corporations outsource narrow constructions of what it means to perform American, embedded in notions of whiteness, Indian call center agents perform a much more nuanced understanding of U.S. culture. Call center agents also narrate the implications of call center work for their personal and cultural lives as they balance the tensions of high paying nighttime employment with familial and cultural relations. It is a delicate negotiation from which emerge performances of postcolonial agencies in a re-colonial context. I analyze these performances for their agency and the oppressions of colonizing corporations to access the cultural costs on both sides of the line

Covariates of turnover intentions of teleworking call center agents in Québec during the COVID-19 pandemic

Les télétravailleurs de plusieurs centres d'appels au Québec ont fourni des données des questionnaires sur leurs diverses demandes au travail (mesurées par les facteurs de stress organisationnels, la charge mentale et la charge émotionnelle), les ressources au travail (mesurées par l'indépendance au travail, la participation et les relations avec les superviseurs) ainsi que pour les mesures des résultats de la satisfaction au travail, de l'engagement organisationnel et de l’intention de quitter. Les hypothèses structurées par le modèle Job Demands-Resources ont été testées à l'aide de méthodes corrélationnelles. Comme prévu, les ressources au travail étaient liées de façon significative à la fois à la satisfaction au travail et à l'engagement organisationnel perçu par l'échantillon. Les demandes au travail prédisaient la satisfaction au travail, mais elles n'étaient pas liées à l'engagement organisationnel. Les implications théoriques et pratiques de ces résultats ont été discutées.Teleworkers from multiple call centers in Québec provided questionnaire data about their various job demands (measured by organizational stressors, mental load, and emotional load), job resources (measured by independence in the work, participation, and relationship with supervisors) as well as for outcome measures of job satisfaction, organizational commitment, and turnover intentions. Hypotheses structured by the JD-R model were tested using correlational methods. As predicted, job resources were significantly related to both job satisfaction and organizational commitment perceived by the sample. Job demands predicted job satisfaction, but they did not relate to organizational commitment. The theoretical and practical implications of these results were discussed