Investigation on Dynamic Speech Emotion from the Perspective of Brain Associative Memory

AbstractMany researchers have studied speech emotion for years from the perspective of psychology to engineering. To date, none has made the speech emotion recognition system intuitive enough in such a way that it can be embedded in automatic answering machines that can effectively detect the various affective states of human verbal communication. In most cases the underlying emotional information was misinterpreted thus resulting in wrong feedbacks and responses. The complexity of understanding and analyzing speech emotion is presented in the dynamics of the emotion itself. Emotion is dynamic and changeable over time. Hence, it is imperative to cater for this parameter to boost the performance of the speech emotion recognition system. In this paper, values of Valence (V) and Arousal(A) are used to generate a recalibrated affective space model. Such approach is adopted from psychologists’ understanding that emotion can be represented using emotion primitives’ values. The VA approach is then coupled with the brain associative memory concept that can provides a better means in understanding the dynamics of speech emotion. Results of such analysis tallies with the psychological findings and has its practical implementation

EEG affective modelling for dysphoria understanding

Dysphoria is a state of dissatisfaction, restlessness or fidgeting. It is a state of feeling unwell in relation to mental and emotional discomfort. If this state is not carefully handled, it may lead to depression, anxiety, and stress. To date, 21-item instruments of Depression, Anxiety and Stress Scale (DASS) is employed to measure dysphoria. Although DASS provides a quantitative assessment of the human affective state, it is subjected to interpretation. To complicate matters, pre-cursor emotion and pre-emotion of the participants can result in biasness of the DASS report. Hence, a more direct method in measuring human affective state by analyzing the brain pattern is proposed. The approach can also address the dynamic affective state which is needed in detecting dysphoria. Brain waves pattern are collected using the electroencephalogram (EEG) device and used as the input to analyze the underlying emotion. In this paper, relevant features were extracted using Mel-frequency cepstral coefficients (MFCC) and classified with Multi-Layer Perceptron (MLP). The experimental results show potential of differentiating between positive and negative emotion with comparable accuracy. Subsequently, it is envisaged that the proposed model can be extended as a tool that can be used to measure stress and anxiety in work places and education institutions

Driver Behaviour State Recognition based on Speech

Researches have linked the cause of traffic accident to driver behavior and some studies provided practical preventive measures based on different input sources. Due to its simplicity to collect, speech can be used as one of the input. The emotion information gathered from speech can be used to measure driver behavior state based on the hypothesis that emotion influences driver behavior. However, the massive amount of driving speech data may hinder optimal performance of processing and analyzing the data due to the computational complexity and time constraint. This paper presents a silence removal approach using Short Term Energy (STE) and Zero Crossing Rate (ZCR) in the pre-processing phase to reduce the unnecessary processing. Mel Frequency Cepstral Coefficient (MFCC) feature extraction method coupled with Multi-Layer Perceptron (MLP) classifier are employed to get the driver behavior state recognition performance. Experimental results demonstrated that the proposed approach can obtain comparable performance with accuracy ranging between 58.7% and 76.6% to differentiate four driver behavior states, namely; talking through mobile phone, laughing, sleepy and normal driving. It is envisaged that such approach can be extended for a more comprehensive driver behavior identification system that may acts as an embedded warning system for sleepy driver

Sensing, interpreting, and anticipating human social behaviour in the real world

Low-level nonverbal social signals like glances, utterances, facial expressions and body language are central to human communicative situations and have been shown to be connected to important high-level constructs, such as emotions, turn-taking, rapport, or leadership. A prerequisite for the creation of social machines that are able to support humans in e.g. education, psychotherapy, or human resources is the ability to automatically sense, interpret, and anticipate human nonverbal behaviour. While promising results have been shown in controlled settings, automatically analysing unconstrained situations, e.g. in daily-life settings, remains challenging. Furthermore, anticipation of nonverbal behaviour in social situations is still largely unexplored. The goal of this thesis is to move closer to the vision of social machines in the real world. It makes fundamental contributions along the three dimensions of sensing, interpreting and anticipating nonverbal behaviour in social interactions. First, robust recognition of low-level nonverbal behaviour lays the groundwork for all further analysis steps. Advancing human visual behaviour sensing is especially relevant as the current state of the art is still not satisfactory in many daily-life situations. While many social interactions take place in groups, current methods for unsupervised eye contact detection can only handle dyadic interactions. We propose a novel unsupervised method for multi-person eye contact detection by exploiting the connection between gaze and speaking turns. Furthermore, we make use of mobile device engagement to address the problem of calibration drift that occurs in daily-life usage of mobile eye trackers. Second, we improve the interpretation of social signals in terms of higher level social behaviours. In particular, we propose the first dataset and method for emotion recognition from bodily expressions of freely moving, unaugmented dyads. Furthermore, we are the first to study low rapport detection in group interactions, as well as investigating a cross-dataset evaluation setting for the emergent leadership detection task. Third, human visual behaviour is special because it functions as a social signal and also determines what a person is seeing at a given moment in time. Being able to anticipate human gaze opens up the possibility for machines to more seamlessly share attention with humans, or to intervene in a timely manner if humans are about to overlook important aspects of the environment. We are the first to propose methods for the anticipation of eye contact in dyadic conversations, as well as in the context of mobile device interactions during daily life, thereby paving the way for interfaces that are able to proactively intervene and support interacting humans.Blick, Gesichtsausdrücke, Körpersprache, oder Prosodie spielen als nonverbale Signale eine zentrale Rolle in menschlicher Kommunikation. Sie wurden durch vielzählige Studien mit wichtigen Konzepten wie Emotionen, Sprecherwechsel, Führung, oder der Qualität des Verhältnisses zwischen zwei Personen in Verbindung gebracht. Damit Menschen effektiv während ihres täglichen sozialen Lebens von Maschinen unterstützt werden können, sind automatische Methoden zur Erkennung, Interpretation, und Antizipation von nonverbalem Verhalten notwendig. Obwohl die bisherige Forschung in kontrollierten Studien zu ermutigenden Ergebnissen gekommen ist, bleibt die automatische Analyse nonverbalen Verhaltens in weniger kontrollierten Situationen eine Herausforderung. Darüber hinaus existieren kaum Untersuchungen zur Antizipation von nonverbalem Verhalten in sozialen Situationen. Das Ziel dieser Arbeit ist, die Vision vom automatischen Verstehen sozialer Situationen ein Stück weit mehr Realität werden zu lassen. Diese Arbeit liefert wichtige Beiträge zur autmatischen Erkennung menschlichen Blickverhaltens in alltäglichen Situationen. Obwohl viele soziale Interaktionen in Gruppen stattfinden, existieren unüberwachte Methoden zur Augenkontakterkennung bisher lediglich für dyadische Interaktionen. Wir stellen einen neuen Ansatz zur Augenkontakterkennung in Gruppen vor, welcher ohne manuelle Annotationen auskommt, indem er sich den statistischen Zusammenhang zwischen Blick- und Sprechverhalten zu Nutze macht. Tägliche Aktivitäten sind eine Herausforderung für Geräte zur mobile Augenbewegungsmessung, da Verschiebungen dieser Geräte zur Verschlechterung ihrer Kalibrierung führen können. In dieser Arbeit verwenden wir Nutzerverhalten an mobilen Endgeräten, um den Effekt solcher Verschiebungen zu korrigieren. Neben der Erkennung verbessert diese Arbeit auch die Interpretation sozialer Signale. Wir veröffentlichen den ersten Datensatz sowie die erste Methode zur Emotionserkennung in dyadischen Interaktionen ohne den Einsatz spezialisierter Ausrüstung. Außerdem stellen wir die erste Studie zur automatischen Erkennung mangelnder Verbundenheit in Gruppeninteraktionen vor, und führen die erste datensatzübergreifende Evaluierung zur Detektion von sich entwickelndem Führungsverhalten durch. Zum Abschluss der Arbeit präsentieren wir die ersten Ansätze zur Antizipation von Blickverhalten in sozialen Interaktionen. Blickverhalten hat die besondere Eigenschaft, dass es sowohl als soziales Signal als auch der Ausrichtung der visuellen Wahrnehmung dient. Somit eröffnet die Fähigkeit zur Antizipation von Blickverhalten Maschinen die Möglichkeit, sich sowohl nahtloser in soziale Interaktionen einzufügen, als auch Menschen zu warnen, wenn diese Gefahr laufen wichtige Aspekte der Umgebung zu übersehen. Wir präsentieren Methoden zur Antizipation von Blickverhalten im Kontext der Interaktion mit mobilen Endgeräten während täglicher Aktivitäten, als auch während dyadischer Interaktionen mittels Videotelefonie

Optimising Emotions, Incubating Falsehoods: How to Protect the Global Civic Body from Disinformation and Misinformation

This open access book deconstructs the core features of online misinformation and disinformation. It finds that the optimisation of emotions for commercial and political gain is a primary cause of false information online. The chapters distil societal harms, evaluate solutions, and consider what must be done to strengthen societies as new biometric forms of emotion profiling emerge. Based on a rich, empirical, and interdisciplinary literature that examines multiple countries, the book will be of interest to scholars and students of Communications, Journalism, Politics, Sociology, Science and Technology Studies, and Information Science, as well as global and local policymakers and ordinary citizens interested in how to prevent the spread of false information worldwide, both now and in the future

Ferocious Logics

Contemporary power manifests in the algorithmic. And yet this power seems incomprehensible: understood as code, it becomes apolitical; understood as a totality, it becomes overwhelming. This book takes an alternate approach, using it to unravel the operations of Uber and Palantir, Airbnb and Amazon Alexa. Moving off the whiteboard and into the world, the algorithmic must negotiate with frictions—the ‘merely’ technical routines of distributing data and running tasks coming together into broader social forces that shape subjectivities, steer bodies, and calibrate relationships. Driven by the imperatives of capital, the algorithmic exhausts subjects and spaces, a double move seeking to both exhaustively apprehend them and exhaust away their productivities. But these on-the-ground encounters also reveal that force is never guaranteed. The irreducibility of the world renders logic inadequate and control gives way to contingency

Ferocious Logics: Unmaking the Algorithm

Contemporary power manifests in the algorithmic. And yet this power seems incomprehensible: understood as code, it becomes apolitical; understood as a totality, it becomes overwhelming. This book takes an alternate approach, using it to unravel the operations of Uber and Palantir, Airbnb and Amazon Alexa. Moving off the whiteboard and into the world, the algorithmic must negotiate with frictions - the 'merely' technical routines of distributing data and running tasks coming together into broader social forces that shape subjectivities, steer bodies, and calibrate relationships. Driven by the imperatives of capital, the algorithmic exhausts subjects and spaces, a double move seeking to both exhaustively apprehend them and exhaust away their productivities. But these on-the-ground encounters also reveal that force is never guaranteed. The irreducibility of the world renders logic inadequate and control gives way to contingency

Optimising Emotions, Incubating Falsehoods

This open access book deconstructs the core features of online misinformation and disinformation. It finds that the optimisation of emotions for commercial and political gain is a primary cause of false information online. The chapters distil societal harms, evaluate solutions, and consider what must be done to strengthen societies as new biometric forms of emotion profiling emerge. Based on a rich, empirical, and interdisciplinary literature that examines multiple countries, the book will be of interest to scholars and students of Communications, Journalism, Politics, Sociology, Science and Technology Studies, and Information Science, as well as global and local policymakers and ordinary citizens interested in how to prevent the spread of false information worldwide, both now and in the future