111 research outputs found
Emotion Recognition by Video: A review
Video emotion recognition is an important branch of affective computing, and
its solutions can be applied in different fields such as human-computer
interaction (HCI) and intelligent medical treatment. Although the number of
papers published in the field of emotion recognition is increasing, there are
few comprehensive literature reviews covering related research on video emotion
recognition. Therefore, this paper selects articles published from 2015 to 2023
to systematize the existing trends in video emotion recognition in related
studies. In this paper, we first talk about two typical emotion models, then we
talk about databases that are frequently utilized for video emotion
recognition, including unimodal databases and multimodal databases. Next, we
look at and classify the specific structure and performance of modern unimodal
and multimodal video emotion recognition methods, talk about the benefits and
drawbacks of each, and then we compare them in detail in the tables. Further,
we sum up the primary difficulties right now looked by video emotion
recognition undertakings and point out probably the most encouraging future
headings, such as establishing an open benchmark database and better multimodal
fusion strategys. The essential objective of this paper is to assist scholarly
and modern scientists with keeping up to date with the most recent advances and
new improvements in this speedy, high-influence field of video emotion
recognition
Confirmation Report: Modelling Interlocutor Confusion in Situated Human Robot Interaction
Human-Robot Interaction (HRI) is an important but challenging field focused on improving the interaction between humans and robots such to make the interaction more intelligent and effective. However, building a natural conversational HRI is an interdisciplinary challenge for scholars, engineers, and designers. It is generally assumed that the pinnacle of human- robot interaction will be having fluid naturalistic conversational interaction that in important ways mimics that of how humans interact with each other. This of course is challenging at a number of levels, and in particular there are considerable difficulties when it comes to naturally monitoring and responding to the user’s mental state. On the topic of mental states, one field that has received little attention to date is moni- toring the user for possible confusion states. Confusion is a non-trivial mental state which can be seen as having at least two substates. There two confusion states can be thought of as being associated with either negative or positive emotions. In the former, when people are productively confused, they have a passion to solve any current difficulties. Meanwhile, people who are in unproductive confusion may lose their engagement and motivation to overcome those difficulties, which in turn may even lead them to drop the current conversation. While there has been some research on confusion monitoring and detection, it has been limited with the most focused on evaluating confusion states in online learning tasks. The central hypothesis of this research is that the monitoring and detection of confusion states in users is essential to fluid task-centric HRI and that it should be possible to detect such confusion and adjust policies to mitigate the confusion in users. In this report, I expand on this hypothesis and set out several research questions. I also provide a comprehensive literature review before outlining work done to date towards my research hypothesis, I also set out plans for future experimental work
Affect Analysis and Membership Recognition in Group Settings
PhD ThesisEmotions play an important role in our day-to-day life in various ways, including, but not
limited to, how we humans communicate and behave. Machines can interact with humans
more naturally and intelligently if they are able to recognise and understand humans’ emotions
and express their own emotions. To achieve this goal, in the past two decades, researchers
have been paying a lot of attention to the analysis of affective states, which has been studied
extensively across various fields, such as neuroscience, psychology, cognitive science, and
computer science. Most of the existing works focus on affect analysis in individual settings,
where there is one person in an image or in a video. However, in the real world, people
are very often with others, or interact in group settings. In this thesis, we will focus on
affect analysis in group settings. Affect analysis in group settings is different from that in
individual settings and provides more challenges due to dynamic interactions between the
group members, various occlusions among people in the scene, and the complex context,
e.g., who people are with, where people are staying and the mutual influences among people
in the group. Because of these challenges, there are still a number of open issues that need
further investigation in order to advance the state of the art, and explore the methodologies
for affect analysis in group settings. These open topics include but are not limited to (1) is
it possible to transfer the methods used for the affect recognition of a person in individual
settings to the affect recognition of each individual in group settings? (2) is it possible to
recognise the affect of one individual using the expressed behaviours of another member in the same group (i.e., cross-subject affect recognition)? (3) can non-verbal behaviours be used
for the recognition of contextual information in group settings?
In this thesis, we investigate the affect analysis in group settings and propose methods to
explore the aforementioned research questions step by step. Firstly, we propose a method for
individual affect recognition in both individual and group videos, which is also used for social
context prediction, i.e., whether a person is alone or within a group. Secondly, we introduce
a novel framework for cross-subject affect analysis in group videos. Specifically, we analyse
the correlation of the affect among group members and investigate the automatic recognition
of the affect of one subject using the behaviours expressed by another subject in the same
group or in a different group. Furthermore, we propose methods for contextual information
prediction in group settings, i.e., group membership recognition - to recognise which group
of the person belongs. Comprehensive experiments are conducted using two datasets that
one contains individual videos and one contains group videos. The experimental results show
that (1) the methods used for affect recognition of a person in individual settings can be
transferred to group settings; (2) the affect of one subject in a group can be better predicted
using the expressive behaviours of another subject within the same group than using that of
a subject from a different group; and (3) contextual information (i.e., whether a person is
staying alone or within a group, and group membership) can be predicted successfully using
non-verbal behaviours
Reconnaissance de l'émotion thermique
Pour améliorer les interactions homme-ordinateur dans les domaines de la santé, de l'e-learning et des jeux vidéos, de nombreux chercheurs ont étudié la reconnaissance des émotions à partir des signaux de texte, de parole, d'expression faciale, de détection d'émotion ou d'électroencéphalographie (EEG). Parmi eux, la reconnaissance d'émotion à l'aide d'EEG a permis une précision satisfaisante. Cependant, le fait d'utiliser des dispositifs d'électroencéphalographie limite la gamme des mouvements de l'utilisateur. Une méthode non envahissante est donc nécessaire pour faciliter la détection des émotions et ses applications. C'est pourquoi nous avons proposé d'utiliser une caméra thermique pour capturer les changements de température de la peau, puis appliquer des algorithmes d'apprentissage machine pour classer les changements d'émotion en conséquence. Cette thèse contient deux études sur la détection d'émotion thermique avec la comparaison de la détection d'émotion basée sur EEG. L'un était de découvrir les profils de détection émotionnelle thermique en comparaison avec la technologie de détection d'émotion basée sur EEG; L'autre était de construire une application avec des algorithmes d'apprentissage en machine profonds pour visualiser la précision et la performance de la détection d'émotion thermique et basée sur EEG. Dans la première recherche, nous avons appliqué HMM dans la reconnaissance de l'émotion thermique, et après avoir comparé à la détection de l'émotion basée sur EEG, nous avons identifié les caractéristiques liées à l'émotion de la température de la peau en termes d'intensité et de rapidité. Dans la deuxième recherche, nous avons mis en place une application de détection d'émotion qui supporte à la fois la détection d'émotion thermique et la détection d'émotion basée sur EEG en appliquant les méthodes d'apprentissage par machine profondes - Réseau Neuronal Convolutif (CNN) et Mémoire à long court-terme (LSTM). La précision de la détection d'émotion basée sur l'image thermique a atteint 52,59% et la précision de la détection basée sur l'EEG a atteint 67,05%. Dans une autre étude, nous allons faire plus de recherches sur l'ajustement des algorithmes d'apprentissage machine pour améliorer la précision de détection d'émotion thermique.To improve computer-human interactions in the areas of healthcare, e-learning and video
games, many researchers have studied on recognizing emotions from text, speech, facial
expressions, emotion detection, or electroencephalography (EEG) signals. Among them,
emotion recognition using EEG has achieved satisfying accuracy. However, wearing
electroencephalography devices limits the range of user movement, thus a noninvasive method
is required to facilitate the emotion detection and its applications. That’s why we proposed using
thermal camera to capture the skin temperature changes and then applying machine learning
algorithms to classify emotion changes accordingly. This thesis contains two studies on thermal
emotion detection with the comparison of EEG-base emotion detection. One was to find out the
thermal emotional detection profiles comparing with EEG-based emotion detection technology;
the other was to implement an application with deep machine learning algorithms to visually
display both thermal and EEG based emotion detection accuracy and performance. In the first
research, we applied HMM in thermal emotion recognition, and after comparing with EEG-base
emotion detection, we identified skin temperature emotion-related features in terms of intensity
and rapidity. In the second research, we implemented an emotion detection application
supporting both thermal emotion detection and EEG-based emotion detection with applying the
deep machine learning methods – Convolutional Neutral Network (CNN) and LSTM (Long-
Short Term Memory). The accuracy of thermal image based emotion detection achieved 52.59%
and the accuracy of EEG based detection achieved 67.05%. In further study, we will do more
research on adjusting machine learning algorithms to improve the thermal emotion detection
precision
Emotion and Stress Recognition Related Sensors and Machine Learning Technologies
This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective
- …